This application is a dog-breed classifier. It takes as input an image and detects if it's an image of either a human or a dog and if it's either one of those then it finds the dog-breed classification that the subject of the image most resembles. If it's neither a human or a dog then it emits an error message. To do this I'm going to try two libraries for each of the human face-detectors and dog detectors and I'm also going to try three Neural Networks to try and classify the dog breeds.

This section does some preliminary set-up for the code that comes later.

I'm going to need a way to tell if an image has a human in it (or not), so I'll build two versions of a detector, one using OpenCV, and one using dlib.

For each detector I'm going to look at an example image before running an assessment of how well it did so I'll select one at random here.

sample_face = numpy.random.choice(human_files, 1)[0]
sample_name = get_name(sample_face)
print(sample_name)

David Anderson

figure, axe = display_image(sample_face, sample_name)

Now I'll take two pre-trained CNNs and use transfer learning to have them detect dogs in images.

Since jupyter (or org-babel) lets you run cells out of sequence I've spent too much time chasing bugs that weren't really bugs, I just hadn't run the right cell. To try and ameliorate that I'm going to use class-based code for the actual implementations.

Although the Inception model does do some classification of dogs, we want an even more fine-tuned model. First I'm going to try to build a naive CNN from scratch, then I'm going to use the Inception model and transfer learning to build a better classifier.

classifier = DogBreedClassifier(model_path=transfer_path)

case = numpy.random.choice(human_files, 1)[0]
classifier(case)

case = numpy.random.choice(dog_files, 1)[0]
classifier(case)

case = "rabbit.jpg"
classifier(case)

Rabbit image from Wikimedia.

case = "hot_dog.jpg"
classifier(case)

The Hot Dog is also from Wikimedia.

case = human_files_short[34]
classifier(case)

So, somehow my class-based detector got smarter than my function based one and can now tell that this isn't a dog…

This is from Udacity's Deep Learning Repository which supports their Deep Learning Nanodegree.

We are going to train a Multi-Layer Perceptron to classify images from the MNIST database of hand-written digits.

We're going to do it using the following steps.

1. Load and visualize the data
2. Define a neural network
3. Train the model
4. Evaluate the performance of our trained model on a test dataset

The first step in a classification task is to take a look at the data, make sure it is loaded in correctly, then make any initial observations about patterns in that data.

The architecture will be responsible for seeing as input a 784-dim Tensor of pixel values for each image, and producing a Tensor of length 10 (our number of classes) that indicates the class scores for an input image. This particular example uses two hidden layers and dropout to avoid overfitting.

These values are based on the keras example implementation.

INPUT_NODES = 28 * 28
HIDDEN_NODES = 512
DROPOUT = 0.2
CLASSES = 10

class Net(nn.Module):
    def __init__(self):
        super().__init__()        
        self.fully_connected_layer_1 = nn.Linear(INPUT_NODES, HIDDEN_NODES)
        self.fully_connected_layer_2 = nn.Linear(HIDDEN_NODES, HIDDEN_NODES)
        self.output = nn.Linear(HIDDEN_NODES, CLASSES)
        self.dropout = nn.Dropout(p=DROPOUT)
        return

    def forward(self, x):
        # flatten image input
        x = x.view(-1, 28 * 28)
        # add hidden layer, with relu activation function
        x = self.dropout(F.relu(self.fully_connected_layer_1(x)))
        x = self.dropout(F.relu(self.fully_connected_layer_2(x)))        
        return self.output(x)

It's recommended that you use cross-entropy loss for classification. If you look at the documentation you can see that PyTorch's cross entropy function applies a softmax function to the output layer and then calculates the log loss (so you don't want to do softmax as part of the model output).

criterion = nn.CrossEntropyLoss()
optimizer = torch.optim.SGD(model.parameters(), lr=0.01)

The steps for training/learning from a batch of data are:

1. Clear the gradients of all optimized variables
2. Forward pass: compute predicted outputs by passing inputs to the model
3. Calculate the loss
4. Backward pass: compute gradient of the loss with respect to model parameters
5. Perform a single optimization step (parameter update)
6. Update average training loss

The following loop trains for 30 epochs; feel free to change this number. For now, we suggest somewhere between 20-50 epochs. As you train, take a look at how the values for the training loss decrease over time. We want it to decrease while also avoiding overfitting the training data.

EPOCHS = 30

start = datetime.now()
model.train() # prep model for training

for epoch in range(EPOCHS):
    # monitor training loss
    train_loss = 0.0
    train_losses = []
    # train the model
    for data, target in train_batches:
        # move it to the GPU or CPU
        data, target = data.to(device), target.to(device)
        # clear the gradients of all optimized variables
        optimizer.zero_grad()
        # forward pass: compute predicted outputs by passing inputs to the model
        output = model(data)
        # calculate the loss
        loss = criterion(output, target)
        # backward pass: compute gradient of the loss with respect to model parameters
        loss.backward()
        # perform a single optimization step (parameter update)
        optimizer.step()
        # update running training loss
        train_loss += loss.item() * data.size(0)

        # print training statistics 
        # calculate average loss over an epoch
    train_loss = train_loss/len(train_batches.dataset)
    train_losses.append(train_loss)
    print('Epoch: {} \tTraining Loss: {:.6f}'.format(
        epoch+1, 
        train_loss
        ))
print("Training Time: {}".format(datetime.now() - start))

Epoch: 1        Training Loss: 0.826836
Epoch: 2        Training Loss: 0.324859
Epoch: 3        Training Loss: 0.251608
Epoch: 4        Training Loss: 0.202294
Epoch: 5        Training Loss: 0.170231
Epoch: 6        Training Loss: 0.146775
Epoch: 7        Training Loss: 0.127352
Epoch: 8        Training Loss: 0.115026
Epoch: 9        Training Loss: 0.104332
Epoch: 10       Training Loss: 0.093575
Epoch: 11       Training Loss: 0.084913
Epoch: 12       Training Loss: 0.077826
Epoch: 13       Training Loss: 0.071506
Epoch: 14       Training Loss: 0.067273
Epoch: 15       Training Loss: 0.063749
Epoch: 16       Training Loss: 0.058150
Epoch: 17       Training Loss: 0.054770
Epoch: 18       Training Loss: 0.051584
Epoch: 19       Training Loss: 0.047762
Epoch: 20       Training Loss: 0.045219
Epoch: 21       Training Loss: 0.041732
Epoch: 22       Training Loss: 0.040526
Epoch: 23       Training Loss: 0.038247
Epoch: 24       Training Loss: 0.035713
Epoch: 25       Training Loss: 0.033801
Epoch: 26       Training Loss: 0.031963
Epoch: 27       Training Loss: 0.031082
Epoch: 28       Training Loss: 0.028971
Epoch: 29       Training Loss: 0.027500
Epoch: 30       Training Loss: 0.026876
Training Time: 0:05:59.808071

Finally, we test our best model on previously unseen test data and evaluate it's performance. Testing on unseen data is a good way to check that our model generalizes well. It may also be useful to be granular in this analysis and take a look at how this model performs on each class as well as looking at its overall loss and accuracy.

In this project we will build a pipeline that can be used within a web or mobile app to process real-world, user-supplied images. Given an image of a dog, our algorithm will identify an estimate of the canine’s breed. If supplied an image of a human, the code will identify the dog breed that the person most resembles.

The dog dataset is in a zip-file hosted on Amazon Web Services. The folder should contain three folders (test, train, and valid) and each of these folders should have 133 folders, one for each dog-breed. It looks like the Stanford Dogs Dataset, but the Stanford data set has 120 breeds, so I don't know the actual source. The human dataset seems to be the Labeled Faces in the Wild data set which was built to study the problem of facial recognition. It's made up of real photos of people taken from the web. Each photo sits in a sub-folder that was given the name of the person (e.g. Michelle_Yeoh). The folder hasn't been split into train-test-validiation folders the way the dog dataset was.

• Unless requested, do not modify code that has already been included.
• In the notebook, you will need to train CNNs in PyTorch. If your CNN is taking too long to train, feel free to pursue one of the options under the section Accelerating the Training Process below.

If your code is taking too long to run, you will need to either reduce the complexity of your chosen CNN architecture or switch to running your code on a GPU. If you'd like to use a GPU, you can spin up an instance of your own:

Your project will be reviewed by a Udacity reviewer against the CNN project rubric. Review this rubric thoroughly and self-evaluate your project before submission. All criteria found in the rubric must meet specifications for you to pass.

Your submission should consist of the github link to your repository. Your repository should contain:

• The dog_app.ipynb file with fully functional code, all code cells executed and displaying output, and all questions answered.
• An HTML or PDF export of the project notebook with the name report.html or report.pdf.

Please do NOT include any of the project data sets provided in the dogImages/ or lfw/ folders.

This is from Udacity's Deep Learning Repository which supports their Deep Learning Nanodegree.

This uses a model trained on ImageNet (available from torchvision) to classify the dataset of cat and dog photos that we used earlier. We're going to use a method called transfer learning where we will use the layers of the pretrained model all the way up until the final classifier which we will define ourselves and train on our new data-set. This way we can take advantage of what the model has already learned for image detection and only train a few layers.

We're going to have to resize the images to be 224x224 to work with the pre-trained models and match the means ([0.485, 0.456, 0.406]) and the standard deviations ([0.229, 0.224, 0.225]) that were used to normalize the original data set.

means = [0.485, 0.456, 0.406]
deviations = [0.229, 0.224, 0.225]
PIXELS = 224

train_transforms = transforms.Compose([transforms.RandomRotation(30),
                                       transforms.RandomResizedCrop(PIXELS),
                                       transforms.RandomHorizontalFlip(),
                                       transforms.ToTensor(),
                                       transforms.Normalize(means,
                                                            deviations)])

test_transforms = transforms.Compose([transforms.Resize(255),
                                      transforms.CenterCrop(PIXELS),
                                      transforms.ToTensor(),
                                      transforms.Normalize(means,
                                                           deviations)])

I'm going to load the DenseNet model.

model = models.densenet121(pretrained=True)

It actually emits a warning that the code is using an incorrect method call somewhere, but I'll ignore that.

UserWarning: nn.init.kaiming_normal is now deprecated in favor of nn.init.kaiming_normal_.
 nn.init.kaiming_normal(m.weight.data)

To speed this up somewhat I'll add (if it's available) a little cuda.

device = torch.device("cuda" if torch.cuda.is_available() else "cpu")

model.to(device)

First we'll set up our criterion - Negative Log Likelihood Loss (NLLLoss) and optimizer - Adam Optimization. Amazingly this only needs one pass through the data set. There's 352 batches in the training data-set so I won't print out each of the outcomes for the epochs.

LEARNING_RATE = 0.003
EPOCHS = 1

criterion = nn.NLLLoss()
optimizer = optim.Adam(model.classifier.parameters(), lr=LEARNING_RATE)

start = datetime.now()
outcome = train_only(model, optimizer, criterion,
                     train_batches,
                     epochs=EPOCHS, emit=False, device=device)
print("Training Time: {}".format(datetime.now() - start))

Training Time: 0:10:35.847469

start = datetime.now()
test_outcome = test_only(model, test_batches, device)
print("Test Time: {}".format(datetime.now() - start))

Test Time: 0:00:46.695136

print(test_outcome)

0.9788

The key bit here was that I was earlier forgetting to add dropout, dropping the accuracy to between .5 and .6.

As the tensors go through the model you should check the shapes to make sure they are correct (or at least what you expect).

• Make sure you are clearing the gradients in the training loop with optimizer.zero_grad()
• In the validation loop, set the network to evaluation mode with model.eval() and then back to training mode with model.train

If you see an error saying pytorch Expected an object of type torch.FloatTensor but found type torch.cuda.FloatTensor then it means something is trying to be run on the CPU but something else wants to use the GPU. Make sure you called .to(device) on the model and all your tensors (including the data).

This is from Udacity's Deep Learning Repository which supports their Deep Learning Nanodegree.

In this notebook, you'll learn how to use pre-trained networks to solved challenging problems in computer vision. Specifically, you'll use networks trained on ImageNet (available from torchvision).

ImageNet is a massive dataset with over 1 million labeled images in 1000 categories. It's used to train deep neural networks using an architecture called convolutional layers. I'm not going to get into the details of convolutional networks here, but if you want to learn more about them, please watch this.

Once trained, these models work astonishingly well as feature detectors for images they weren't trained on. Using a pre-trained network on images not in the training set is called transfer learning. Here we'll use transfer learning to train a network that can classify our cat and dog photos with near perfect accuracy.

With torchvision.models you can download these pre-trained networks and use them in your applications. We'll include models in our imports now.

Most of the pretrained models require the input to be 224x224 images. Also, we'll need to match the normalization used when the models were trained. Each color channel was normalized separately, the means are [0.485, 0.456, 0.406] and the standard deviations are [0.229, 0.224, 0.225].

means = [0.485, 0.456, 0.406]
deviations = [0.229, 0.224, 0.225]

train_transforms = transforms.Compose([transforms.RandomRotation(30),
                                       transforms.RandomResizedCrop(224),
                                       transforms.RandomHorizontalFlip(),
                                       transforms.ToTensor(),
                                       transforms.Normalize(means,
                                                            deviations)])

test_transforms = transforms.Compose([transforms.Resize(255),
                                      transforms.CenterCrop(224),
                                      transforms.ToTensor(),
                                      transforms.Normalize(means,
                                                           deviations)])

We are going to load the DenseNet model.

model = models.densenet121(pretrained=True)

print(model)

DenseNet(
  (features): Sequential(
    (conv0): Conv2d(3, 64, kernel_size=(7, 7), stride=(2, 2), padding=(3, 3), bias=False)
    (norm0): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
    (relu0): ReLU(inplace)
    (pool0): MaxPool2d(kernel_size=3, stride=2, padding=1, dilation=1, ceil_mode=False)
    (denseblock1): _DenseBlock(
      (denselayer1): _DenseLayer(
        (norm1): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        (relu1): ReLU(inplace)
        (conv1): Conv2d(64, 128, kernel_size=(1, 1), stride=(1, 1), bias=False)
        (norm2): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        (relu2): ReLU(inplace)
        (conv2): Conv2d(128, 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
      )
      (denselayer2): _DenseLayer(
        (norm1): BatchNorm2d(96, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        (relu1): ReLU(inplace)
        (conv1): Conv2d(96, 128, kernel_size=(1, 1), stride=(1, 1), bias=False)
        (norm2): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        (relu2): ReLU(inplace)
        (conv2): Conv2d(128, 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
      )
      (denselayer3): _DenseLayer(
        (norm1): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        (relu1): ReLU(inplace)
        (conv1): Conv2d(128, 128, kernel_size=(1, 1), stride=(1, 1), bias=False)
        (norm2): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        (relu2): ReLU(inplace)
        (conv2): Conv2d(128, 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
      )
      (denselayer4): _DenseLayer(
        (norm1): BatchNorm2d(160, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        (relu1): ReLU(inplace)
        (conv1): Conv2d(160, 128, kernel_size=(1, 1), stride=(1, 1), bias=False)
        (norm2): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        (relu2): ReLU(inplace)
        (conv2): Conv2d(128, 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
      )
      (denselayer5): _DenseLayer(
        (norm1): BatchNorm2d(192, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        (relu1): ReLU(inplace)
        (conv1): Conv2d(192, 128, kernel_size=(1, 1), stride=(1, 1), bias=False)
        (norm2): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        (relu2): ReLU(inplace)
        (conv2): Conv2d(128, 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
      )
      (denselayer6): _DenseLayer(
        (norm1): BatchNorm2d(224, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        (relu1): ReLU(inplace)
        (conv1): Conv2d(224, 128, kernel_size=(1, 1), stride=(1, 1), bias=False)
        (norm2): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        (relu2): ReLU(inplace)
        (conv2): Conv2d(128, 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
      )
    )
    (transition1): _Transition(
      (norm): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
      (relu): ReLU(inplace)
      (conv): Conv2d(256, 128, kernel_size=(1, 1), stride=(1, 1), bias=False)
      (pool): AvgPool2d(kernel_size=2, stride=2, padding=0)
    )
    (denseblock2): _DenseBlock(
      (denselayer1): _DenseLayer(
        (norm1): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        (relu1): ReLU(inplace)
        (conv1): Conv2d(128, 128, kernel_size=(1, 1), stride=(1, 1), bias=False)
        (norm2): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        (relu2): ReLU(inplace)
        (conv2): Conv2d(128, 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
      )
      (denselayer2): _DenseLayer(
        (norm1): BatchNorm2d(160, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        (relu1): ReLU(inplace)
        (conv1): Conv2d(160, 128, kernel_size=(1, 1), stride=(1, 1), bias=False)
        (norm2): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        (relu2): ReLU(inplace)
        (conv2): Conv2d(128, 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
      )
      (denselayer3): _DenseLayer(
        (norm1): BatchNorm2d(192, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        (relu1): ReLU(inplace)
        (conv1): Conv2d(192, 128, kernel_size=(1, 1), stride=(1, 1), bias=False)
        (norm2): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        (relu2): ReLU(inplace)
        (conv2): Conv2d(128, 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
      )
      (denselayer4): _DenseLayer(
        (norm1): BatchNorm2d(224, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        (relu1): ReLU(inplace)
        (conv1): Conv2d(224, 128, kernel_size=(1, 1), stride=(1, 1), bias=False)
        (norm2): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        (relu2): ReLU(inplace)
        (conv2): Conv2d(128, 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
      )
      (denselayer5): _DenseLayer(
        (norm1): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        (relu1): ReLU(inplace)
        (conv1): Conv2d(256, 128, kernel_size=(1, 1), stride=(1, 1), bias=False)
        (norm2): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        (relu2): ReLU(inplace)
        (conv2): Conv2d(128, 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
      )
      (denselayer6): _DenseLayer(
        (norm1): BatchNorm2d(288, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        (relu1): ReLU(inplace)
        (conv1): Conv2d(288, 128, kernel_size=(1, 1), stride=(1, 1), bias=False)
        (norm2): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        (relu2): ReLU(inplace)
        (conv2): Conv2d(128, 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
      )
      (denselayer7): _DenseLayer(
        (norm1): BatchNorm2d(320, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        (relu1): ReLU(inplace)
        (conv1): Conv2d(320, 128, kernel_size=(1, 1), stride=(1, 1), bias=False)
        (norm2): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        (relu2): ReLU(inplace)
        (conv2): Conv2d(128, 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
      )
      (denselayer8): _DenseLayer(
        (norm1): BatchNorm2d(352, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        (relu1): ReLU(inplace)
        (conv1): Conv2d(352, 128, kernel_size=(1, 1), stride=(1, 1), bias=False)
        (norm2): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        (relu2): ReLU(inplace)
        (conv2): Conv2d(128, 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
      )
      (denselayer9): _DenseLayer(
        (norm1): BatchNorm2d(384, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        (relu1): ReLU(inplace)
        (conv1): Conv2d(384, 128, kernel_size=(1, 1), stride=(1, 1), bias=False)
        (norm2): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        (relu2): ReLU(inplace)
        (conv2): Conv2d(128, 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
      )
      (denselayer10): _DenseLayer(
        (norm1): BatchNorm2d(416, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        (relu1): ReLU(inplace)
        (conv1): Conv2d(416, 128, kernel_size=(1, 1), stride=(1, 1), bias=False)
        (norm2): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        (relu2): ReLU(inplace)
        (conv2): Conv2d(128, 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
      )
      (denselayer11): _DenseLayer(
        (norm1): BatchNorm2d(448, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        (relu1): ReLU(inplace)
        (conv1): Conv2d(448, 128, kernel_size=(1, 1), stride=(1, 1), bias=False)
        (norm2): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        (relu2): ReLU(inplace)
        (conv2): Conv2d(128, 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
      )
      (denselayer12): _DenseLayer(
        (norm1): BatchNorm2d(480, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        (relu1): ReLU(inplace)
        (conv1): Conv2d(480, 128, kernel_size=(1, 1), stride=(1, 1), bias=False)
        (norm2): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        (relu2): ReLU(inplace)
        (conv2): Conv2d(128, 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
      )
    )
    (transition2): _Transition(
      (norm): BatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
      (relu): ReLU(inplace)
      (conv): Conv2d(512, 256, kernel_size=(1, 1), stride=(1, 1), bias=False)
      (pool): AvgPool2d(kernel_size=2, stride=2, padding=0)
    )
    (denseblock3): _DenseBlock(
      (denselayer1): _DenseLayer(
        (norm1): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        (relu1): ReLU(inplace)
        (conv1): Conv2d(256, 128, kernel_size=(1, 1), stride=(1, 1), bias=False)
        (norm2): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        (relu2): ReLU(inplace)
        (conv2): Conv2d(128, 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
      )
      (denselayer2): _DenseLayer(
        (norm1): BatchNorm2d(288, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        (relu1): ReLU(inplace)
        (conv1): Conv2d(288, 128, kernel_size=(1, 1), stride=(1, 1), bias=False)
        (norm2): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        (relu2): ReLU(inplace)
        (conv2): Conv2d(128, 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
      )
      (denselayer3): _DenseLayer(
        (norm1): BatchNorm2d(320, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        (relu1): ReLU(inplace)
        (conv1): Conv2d(320, 128, kernel_size=(1, 1), stride=(1, 1), bias=False)
        (norm2): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        (relu2): ReLU(inplace)
        (conv2): Conv2d(128, 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
      )
      (denselayer4): _DenseLayer(
        (norm1): BatchNorm2d(352, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        (relu1): ReLU(inplace)
        (conv1): Conv2d(352, 128, kernel_size=(1, 1), stride=(1, 1), bias=False)
        (norm2): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        (relu2): ReLU(inplace)
        (conv2): Conv2d(128, 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
      )
      (denselayer5): _DenseLayer(
        (norm1): BatchNorm2d(384, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        (relu1): ReLU(inplace)
        (conv1): Conv2d(384, 128, kernel_size=(1, 1), stride=(1, 1), bias=False)
        (norm2): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        (relu2): ReLU(inplace)
        (conv2): Conv2d(128, 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
      )
      (denselayer6): _DenseLayer(
        (norm1): BatchNorm2d(416, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        (relu1): ReLU(inplace)
        (conv1): Conv2d(416, 128, kernel_size=(1, 1), stride=(1, 1), bias=False)
        (norm2): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        (relu2): ReLU(inplace)
        (conv2): Conv2d(128, 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
      )
      (denselayer7): _DenseLayer(
        (norm1): BatchNorm2d(448, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        (relu1): ReLU(inplace)
        (conv1): Conv2d(448, 128, kernel_size=(1, 1), stride=(1, 1), bias=False)
        (norm2): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        (relu2): ReLU(inplace)
        (conv2): Conv2d(128, 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
      )
      (denselayer8): _DenseLayer(
        (norm1): BatchNorm2d(480, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        (relu1): ReLU(inplace)
        (conv1): Conv2d(480, 128, kernel_size=(1, 1), stride=(1, 1), bias=False)
        (norm2): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        (relu2): ReLU(inplace)
        (conv2): Conv2d(128, 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
      )
      (denselayer9): _DenseLayer(
        (norm1): BatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        (relu1): ReLU(inplace)
        (conv1): Conv2d(512, 128, kernel_size=(1, 1), stride=(1, 1), bias=False)
        (norm2): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        (relu2): ReLU(inplace)
        (conv2): Conv2d(128, 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
      )
      (denselayer10): _DenseLayer(
        (norm1): BatchNorm2d(544, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        (relu1): ReLU(inplace)
        (conv1): Conv2d(544, 128, kernel_size=(1, 1), stride=(1, 1), bias=False)
        (norm2): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        (relu2): ReLU(inplace)
        (conv2): Conv2d(128, 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
      )
      (denselayer11): _DenseLayer(
        (norm1): BatchNorm2d(576, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        (relu1): ReLU(inplace)
        (conv1): Conv2d(576, 128, kernel_size=(1, 1), stride=(1, 1), bias=False)
        (norm2): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        (relu2): ReLU(inplace)
        (conv2): Conv2d(128, 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
      )
      (denselayer12): _DenseLayer(
        (norm1): BatchNorm2d(608, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        (relu1): ReLU(inplace)
        (conv1): Conv2d(608, 128, kernel_size=(1, 1), stride=(1, 1), bias=False)
        (norm2): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        (relu2): ReLU(inplace)
        (conv2): Conv2d(128, 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
      )
      (denselayer13): _DenseLayer(
        (norm1): BatchNorm2d(640, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        (relu1): ReLU(inplace)
        (conv1): Conv2d(640, 128, kernel_size=(1, 1), stride=(1, 1), bias=False)
        (norm2): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        (relu2): ReLU(inplace)
        (conv2): Conv2d(128, 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
      )
      (denselayer14): _DenseLayer(
        (norm1): BatchNorm2d(672, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        (relu1): ReLU(inplace)
        (conv1): Conv2d(672, 128, kernel_size=(1, 1), stride=(1, 1), bias=False)
        (norm2): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        (relu2): ReLU(inplace)
        (conv2): Conv2d(128, 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
      )
      (denselayer15): _DenseLayer(
        (norm1): BatchNorm2d(704, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        (relu1): ReLU(inplace)
        (conv1): Conv2d(704, 128, kernel_size=(1, 1), stride=(1, 1), bias=False)
        (norm2): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        (relu2): ReLU(inplace)
        (conv2): Conv2d(128, 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
      )
      (denselayer16): _DenseLayer(
        (norm1): BatchNorm2d(736, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        (relu1): ReLU(inplace)
        (conv1): Conv2d(736, 128, kernel_size=(1, 1), stride=(1, 1), bias=False)
        (norm2): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        (relu2): ReLU(inplace)
        (conv2): Conv2d(128, 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
      )
      (denselayer17): _DenseLayer(
        (norm1): BatchNorm2d(768, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        (relu1): ReLU(inplace)
        (conv1): Conv2d(768, 128, kernel_size=(1, 1), stride=(1, 1), bias=False)
        (norm2): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        (relu2): ReLU(inplace)
        (conv2): Conv2d(128, 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
      )
      (denselayer18): _DenseLayer(
        (norm1): BatchNorm2d(800, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        (relu1): ReLU(inplace)
        (conv1): Conv2d(800, 128, kernel_size=(1, 1), stride=(1, 1), bias=False)
        (norm2): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        (relu2): ReLU(inplace)
        (conv2): Conv2d(128, 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
      )
      (denselayer19): _DenseLayer(
        (norm1): BatchNorm2d(832, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        (relu1): ReLU(inplace)
        (conv1): Conv2d(832, 128, kernel_size=(1, 1), stride=(1, 1), bias=False)
        (norm2): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        (relu2): ReLU(inplace)
        (conv2): Conv2d(128, 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
      )
      (denselayer20): _DenseLayer(
        (norm1): BatchNorm2d(864, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        (relu1): ReLU(inplace)
        (conv1): Conv2d(864, 128, kernel_size=(1, 1), stride=(1, 1), bias=False)
        (norm2): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        (relu2): ReLU(inplace)
        (conv2): Conv2d(128, 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
      )
      (denselayer21): _DenseLayer(
        (norm1): BatchNorm2d(896, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        (relu1): ReLU(inplace)
        (conv1): Conv2d(896, 128, kernel_size=(1, 1), stride=(1, 1), bias=False)
        (norm2): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        (relu2): ReLU(inplace)
        (conv2): Conv2d(128, 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
      )
      (denselayer22): _DenseLayer(
        (norm1): BatchNorm2d(928, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        (relu1): ReLU(inplace)
        (conv1): Conv2d(928, 128, kernel_size=(1, 1), stride=(1, 1), bias=False)
        (norm2): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        (relu2): ReLU(inplace)
        (conv2): Conv2d(128, 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
      )
      (denselayer23): _DenseLayer(
        (norm1): BatchNorm2d(960, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        (relu1): ReLU(inplace)
        (conv1): Conv2d(960, 128, kernel_size=(1, 1), stride=(1, 1), bias=False)
        (norm2): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        (relu2): ReLU(inplace)
        (conv2): Conv2d(128, 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
      )
      (denselayer24): _DenseLayer(
        (norm1): BatchNorm2d(992, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        (relu1): ReLU(inplace)
        (conv1): Conv2d(992, 128, kernel_size=(1, 1), stride=(1, 1), bias=False)
        (norm2): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        (relu2): ReLU(inplace)
        (conv2): Conv2d(128, 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
      )
    )
    (transition3): _Transition(
      (norm): BatchNorm2d(1024, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
      (relu): ReLU(inplace)
      (conv): Conv2d(1024, 512, kernel_size=(1, 1), stride=(1, 1), bias=False)
      (pool): AvgPool2d(kernel_size=2, stride=2, padding=0)
    )
    (denseblock4): _DenseBlock(
      (denselayer1): _DenseLayer(
        (norm1): BatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        (relu1): ReLU(inplace)
        (conv1): Conv2d(512, 128, kernel_size=(1, 1), stride=(1, 1), bias=False)
        (norm2): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        (relu2): ReLU(inplace)
        (conv2): Conv2d(128, 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
      )
      (denselayer2): _DenseLayer(
        (norm1): BatchNorm2d(544, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        (relu1): ReLU(inplace)
        (conv1): Conv2d(544, 128, kernel_size=(1, 1), stride=(1, 1), bias=False)
        (norm2): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        (relu2): ReLU(inplace)
        (conv2): Conv2d(128, 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
      )
      (denselayer3): _DenseLayer(
        (norm1): BatchNorm2d(576, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        (relu1): ReLU(inplace)
        (conv1): Conv2d(576, 128, kernel_size=(1, 1), stride=(1, 1), bias=False)
        (norm2): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        (relu2): ReLU(inplace)
        (conv2): Conv2d(128, 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
      )
      (denselayer4): _DenseLayer(
        (norm1): BatchNorm2d(608, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        (relu1): ReLU(inplace)
        (conv1): Conv2d(608, 128, kernel_size=(1, 1), stride=(1, 1), bias=False)
        (norm2): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        (relu2): ReLU(inplace)
        (conv2): Conv2d(128, 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
      )
      (denselayer5): _DenseLayer(
        (norm1): BatchNorm2d(640, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        (relu1): ReLU(inplace)
        (conv1): Conv2d(640, 128, kernel_size=(1, 1), stride=(1, 1), bias=False)
        (norm2): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        (relu2): ReLU(inplace)
        (conv2): Conv2d(128, 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
      )
      (denselayer6): _DenseLayer(
        (norm1): BatchNorm2d(672, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        (relu1): ReLU(inplace)
        (conv1): Conv2d(672, 128, kernel_size=(1, 1), stride=(1, 1), bias=False)
        (norm2): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        (relu2): ReLU(inplace)
        (conv2): Conv2d(128, 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
      )
      (denselayer7): _DenseLayer(
        (norm1): BatchNorm2d(704, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        (relu1): ReLU(inplace)
        (conv1): Conv2d(704, 128, kernel_size=(1, 1), stride=(1, 1), bias=False)
        (norm2): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        (relu2): ReLU(inplace)
        (conv2): Conv2d(128, 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
      )
      (denselayer8): _DenseLayer(
        (norm1): BatchNorm2d(736, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        (relu1): ReLU(inplace)
        (conv1): Conv2d(736, 128, kernel_size=(1, 1), stride=(1, 1), bias=False)
        (norm2): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        (relu2): ReLU(inplace)
        (conv2): Conv2d(128, 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
      )
      (denselayer9): _DenseLayer(
        (norm1): BatchNorm2d(768, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        (relu1): ReLU(inplace)
        (conv1): Conv2d(768, 128, kernel_size=(1, 1), stride=(1, 1), bias=False)
        (norm2): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        (relu2): ReLU(inplace)
        (conv2): Conv2d(128, 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
      )
      (denselayer10): _DenseLayer(
        (norm1): BatchNorm2d(800, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        (relu1): ReLU(inplace)
        (conv1): Conv2d(800, 128, kernel_size=(1, 1), stride=(1, 1), bias=False)
        (norm2): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        (relu2): ReLU(inplace)
        (conv2): Conv2d(128, 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
      )
      (denselayer11): _DenseLayer(
        (norm1): BatchNorm2d(832, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        (relu1): ReLU(inplace)
        (conv1): Conv2d(832, 128, kernel_size=(1, 1), stride=(1, 1), bias=False)
        (norm2): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        (relu2): ReLU(inplace)
        (conv2): Conv2d(128, 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
      )
      (denselayer12): _DenseLayer(
        (norm1): BatchNorm2d(864, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        (relu1): ReLU(inplace)
        (conv1): Conv2d(864, 128, kernel_size=(1, 1), stride=(1, 1), bias=False)
        (norm2): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        (relu2): ReLU(inplace)
        (conv2): Conv2d(128, 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
      )
      (denselayer13): _DenseLayer(
        (norm1): BatchNorm2d(896, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        (relu1): ReLU(inplace)
        (conv1): Conv2d(896, 128, kernel_size=(1, 1), stride=(1, 1), bias=False)
        (norm2): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        (relu2): ReLU(inplace)
        (conv2): Conv2d(128, 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
      )
      (denselayer14): _DenseLayer(
        (norm1): BatchNorm2d(928, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        (relu1): ReLU(inplace)
        (conv1): Conv2d(928, 128, kernel_size=(1, 1), stride=(1, 1), bias=False)
        (norm2): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        (relu2): ReLU(inplace)
        (conv2): Conv2d(128, 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
      )
      (denselayer15): _DenseLayer(
        (norm1): BatchNorm2d(960, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        (relu1): ReLU(inplace)
        (conv1): Conv2d(960, 128, kernel_size=(1, 1), stride=(1, 1), bias=False)
        (norm2): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        (relu2): ReLU(inplace)
        (conv2): Conv2d(128, 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
      )
      (denselayer16): _DenseLayer(
        (norm1): BatchNorm2d(992, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        (relu1): ReLU(inplace)
        (conv1): Conv2d(992, 128, kernel_size=(1, 1), stride=(1, 1), bias=False)
        (norm2): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        (relu2): ReLU(inplace)
        (conv2): Conv2d(128, 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
      )
    )
    (norm5): BatchNorm2d(1024, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
  )
  (classifier): Linear(in_features=1024, out_features=1000, bias=True)
)

This model is built out of two main parts, the features and the classifier. The features part is a stack of convolutional layers and overall works as a feature detector that can be fed into a classifier. The classifier part is a single fully-connected layer (classifier): Linear(in_features=1024, out_features=1000). This layer was trained on the ImageNet dataset, so it won't work for our specific problem. That means we need to replace the classifier, but the features will work perfectly on their own. In general, I think about pre-trained networks as amazingly good feature detectors that can be used as the input for simple feed-forward classifiers.

Next we want to freeze the parameters so we don't backprop through them.

for param in model.parameters():
    param.requires_grad = False

And now we build our classifier model.

classifier = nn.Sequential(OrderedDict([
                          ('fc1', nn.Linear(1024, 500)),
                          ('relu', nn.ReLU()),
                          ('fc2', nn.Linear(500, 2)),
                          ('output', nn.LogSoftmax(dim=1))
                          ]))

With our model built, we need to train the classifier. However, now we're using a really deep neural network. If you try to train this on a CPU like normal, it will take a long, long time. Instead, we're going to use the GPU to do the calculations. The linear algebra computations are done in parallel on the GPU leading to 100x increased training speeds. It's also possible to train on multiple GPUs, further decreasing training time.

PyTorch, along with pretty much every other deep learning framework, uses CUDA to efficiently compute the forward and backwards passes on the GPU. In PyTorch, you move your model parameters and other tensors to the GPU memory using model.to('cuda'). You can move them back from the GPU with model.to('cpu') which you'll commonly do when you need to operate on the network output outside of PyTorch. As a demonstration of the increased speed, I'll compare how long it takes to perform a forward and backward pass with and without a GPU.

criterion = nn.NLLLoss()
device = "cpu"
# Only train the classifier parameters, feature parameters are frozen
optimizer = optim.Adam(model.classifier.parameters(), lr=0.003)
model.to(device)

for index, (inputs, labels) in enumerate(train_batches):
    # Move input and label tensors to the GPU
    inputs, labels = inputs.to(device), labels.to(device)

    start = datetime.now()

    outputs = model.forward(inputs)
    loss = criterion(outputs, labels)
    loss.backward()
    optimizer.step()

    if index==3:
        break

print("Device = {}; Time per batch: {} seconds".format(
    device, (datetime.now() - start)/3
    ))

Device = cpu; Time per batch: 0:00:12.372973 seconds

device = "cuda"
criterion = nn.NLLLoss()
# Only train the classifier parameters, feature parameters are frozen
optimizer = optim.Adam(model.classifier.parameters(), lr=0.003)
model.to(device)

for index, (inputs, labels) in enumerate(train_batches):
    # Move input and label tensors to the GPU
    inputs, labels = inputs.to(device), labels.to(device)

    start = datetime.now()

    outputs = model.forward(inputs)
    loss = criterion(outputs, labels)
    loss.backward()
    optimizer.step()

    if index==3:
        break

print("Device = {}; Time per batch: {} seconds".format(
        device, (datetime.now() - start)/3
))

Device = cuda; Time per batch: 0:00:00.008037 seconds

So, it takes less than a second compared to 12 seconds. Interestingly, I kept getting a CUDA out of memory error when I had seaborn and matplotlib imported at the top. I don't know what the conflict is, but it's something to watch out for.

You can write device agnostic code which will automatically use CUDA if it's enabled like so at the beginning of your code:

device = torch.device("cuda:0" if torch.cuda.is_available() else "cpu")

Then whenever you get a new Tensor or Module it won't copy if they are already on the desired device (it will just return the original object).

input = data.to(device)
model = MyModule(...).to(device)

First a short test to make sure this works.

train_iter = iter(train_batches)
train_small = [train_iter.next() for item in range(2)]
test_iter = iter(test_batches)
test_small = [test_iter.next() for item in range(2)]
outcome = train(model, optimizer, criterion, train_small, test_small, epochs=1, device="cuda")

Epoch: 1/30 Training loss: 0.43 Test Loss: 2.63 Test Accuracy: 0.56

Okay, so now for a long one. Time to get some coffee.

outcome = train_only(model, optimizer, criterion, train_batches,
                     epochs=10, device=device)
torch.save(model.state_dict(), "cat_dog_model.pth")

test_outcome = test_only(model, criterion, test_batches, devicej)

print(test_outcome.iloc[-1])

Test Loss        1.532174
Test Accuracy    0.630859
Name: 39, dtype: float64

So, it hasn't actually gotten better, if anything it got worse. Does this mean it's overfitting?

I peeked at the solution notebook and it has fewer nodes in the first linear layer and adds dropout. Interestingly the lecture has more nodes in the first layer, but I'll try fewer first.

This is from Udacity's Deep Learning Repository which supports their Deep Learning Nanodegree.

So far we've been working with fairly artificial datasets that you wouldn't typically be using in real projects (28 x 28 pixels is very low resolution). Instead, you'll likely be dealing with full-sized images like you'd get from cameras. In this notebook, we'll look at how to load images and use them to train neural networks.

We'll be using a dataset of cat and dog photos available from Kaggle that was created to test whether a machine would be able to defeat the Asirra CAPTCHA system by identifying whether an image had a cat or a dog.

We'll use this dataset to train a neural network that can differentiate between cats and dogs. These days it doesn't seem like a big accomplishment, but five years ago it was a serious challenge for computer vision systems.

The easiest way to load image data is with datasets.ImageFolder from torchvision. In general you'll use ImageFolder like so:

dataset = datasets.ImageFolder('path/to/data', transform=transforms)

where path/to/data is the file path to the data directory and transforms is a list of processing steps built with the transforms module from torchvision. ImageFolder expects the files and directories to be constructed like so:

root/dog/xxx.png
root/dog/xxy.png
root/dog/xxz.png

root/cat/123.png
root/cat/nsdf3.png
root/cat/asd932_.png

where each class has it's own directory (cat and dog) for the images. The images are then labeled with the class taken from the directory name. So here, the image 123.png would be loaded with the class label cat. You can download the dataset already structured like this from here. I've also split it into a training set and test set (note that the data-set is almost 600 Megabytes so make sure you have broadband if you want to download it).

A common strategy for training neural networks is to introduce randomness in the input data itself. For example, you can randomly rotate, mirror, scale, and/or crop your images during training. This will help your network generalize as it's seeing the same images but in different locations, with different sizes, in different orientations, etc.

To randomly rotate, scale and crop, then flip your images you would define your transforms like this:

train_transforms = transforms.Compose([transforms.RandomRotation(30),
                                       transforms.RandomResizedCrop(100),
                                       transforms.RandomHorizontalFlip(),
                                       transforms.ToTensor(),
                                       transforms.Normalize([0.5, 0.5, 0.5], 
                                                            [0.5, 0.5, 0.5])])

You'll also typically want to normalize images with transforms.Normalize. You pass in a list of means and list of standard deviations, then the color channels are normalized like so

input[channel] = (input[channel] - mean[channel]) / std[channel]

Subtracting mean centers the data around zero and dividing by std squishes the values to be between -1 and 1. Normalizing helps keep the network work weights near zero which in turn makes backpropagation more stable. Without normalization, networks will tend to fail to learn.

You can find a list of all the available transforms here . When you're testing however, you'll want to use images that aren't altered (except you'll need to normalize the same way). So, for validation/test images, you'll typically just resize and crop.

The Training Transformations:

• RandomRotation: takes the maximum number of degrees to rotate the image
• RandomResizedCrop: scales and crops the image - we're only passing in the expected output size
• RandomHorizontalFlip: 50-50 chance that the image will be flipped horizontally.

means = deviations = [0.5, 0.5, 0.5]
train_transforms = transforms.Compose([transforms.RandomRotation(30),
                                       transforms.RandomResizedCrop(100),
                                       transforms.RandomHorizontalFlip(),
                                       transforms.ToTensor(),
                                       transforms.Normalize(means, 
                                                            deviations)])

test_transforms = transforms.Compose([transforms.Resize(255),
                                      transforms.CenterCrop(224),
                                      transforms.ToTensor(),
                                      transforms.Normalize(mean=means,
                                                           std=deviations)])

Now we create the testing and training data. Although I loaded the training data before, I didn't apply all the extra transforms so I'm going to re-load it

test_path = DataPathTwo(folder_key="CAT_DOG_TEST")
train_data = datasets.ImageFolder(train_path.folder, transform=train_transforms)
test_data = datasets.ImageFolder(test_path.folder, transform=test_transforms)

train_batches = torch.utils.data.DataLoader(train_data, batch_size=32)
test_batches = torch.utils.data.DataLoader(test_data, batch_size=32)

Here are the first four images in the training set after they were transformed.

images, labels = iter(train_batches).next()
fig, axes = pyplot.subplots(figsize=(10,4), ncols=4)
for index in range(4):
    ax = axes[index]
    helper.imshow(images[index], ax=ax)

At this point you should be able to load data for training and testing. Now, you should try building a network that can classify cats vs dogs. This is quite a bit more complicated than before with the MNIST and Fashion-MNIST datasets. To be honest, you probably won't get it to work with a fully-connected network, no matter how deep. These images have three color channels and at a higher resolution (so far you've seen 28x28 images which are tiny).

I'm just going to try and apply the Dropout Model from the FASHION-MNIST examples and see what happens. But, it turns out that the input shapes are wrong. Each image is a (3, 100, 100) tensor.

parameters = HyperParameters()
parameters.inputs = 3 * 100 * 100
parameters.outputs = 2

model = DropoutModel(parameters)
criterion = nn.NLLLoss()
optimizer = optim.Adam(model.parameters(), lr=HyperParameters.learning_rate)
outcomes = train(model=model,
                 optimizer=optimizer,
                 criterion=criterion,
                 train_batches=train_batches,
                 test_batches=test_batches)

Okay, this doesn't work, there's a mismatched size problem that I can't figure out. Maybe I'll come back to this.

This is from Udacity's Deep Learning Repository which supports their Deep Learning Nanodegree.

In this notebook we're going to look at how to save and load models with PyTorch.

Once again we're going to use the fashion-MNIST data.

I'm re-using the DropoutModel from the previous lesson about avoiding over-fitting using dropout. I'm also re-using the (somewhat updated) train function.

model = DropoutModel()
criterion = nn.NLLLoss()
optimizer = optim.Adam(model.parameters(), lr=0.001)

train(model=model, optimizer=optimizer, criterion=criterion,
      train_batches=training, test_batches=testing, epochs=2)

Epoch: 1/30 Training loss: 2.41 Test Loss: 2.40 Test Accuracy: 0.09
Epoch: 2/30 Training loss: 2.41 Test Loss: 2.40 Test Accuracy: 0.09

Rather than re-training your model every time you want to use it you can instead save it an re-load the pre-trained model when you need it.

The parameters for PyTorch networks are stored in a model's state_dict.

print("Our model: \n\n", model, '\n')
print("The state dict keys: \n\n", model.state_dict().keys())

Our model: 

 DropoutModel(
  (input_to_hidden): Linear(in_features=784, out_features=256, bias=True)
  (hidden_1_to_hidden_2): Linear(in_features=256, out_features=128, bias=True)
  (hidden_2_to_hidden_3): Linear(in_features=128, out_features=64, bias=True)
  (hidden_3_to_output): Linear(in_features=64, out_features=10, bias=True)
  (dropout): Dropout(p=0.2)
) 

The state dict keys: 

 odict_keys(['input_to_hidden.weight', 'input_to_hidden.bias', 'hidden_1_to_hidden_2.weight', 'hidden_1_to_hidden_2.bias', 'hidden_2_to_hidden_3.weight', 'hidden_2_to_hidden_3.bias', 'hidden_3_to_output.weight', 'hidden_3_to_output.bias'])

The simplest thing to do is simply save the state dict with torch.save, which uses python's pickle to serialze the settings. PyTorch has an explanation for why you would prefer saving the settings instead of the entire model.

As an example, we can save our trained model's settings to a file checkpoint.pth.

file_name = "checkpoint.pth"
torch.save(model.state_dict(), file_name)

check_path = Path(file_name)
print("File Size: {} K".format(check_path.stat().st_size/10**3))

File Size: 972.392 K

So it's almost a megabyte, better remember to clean it up later.

I couldn't find an explanation for the file-extension, but the pytorch documentation mentions that it's a convention to use .pt and .pth as extensions. I'm assuming pt is for PyTorch and the h is for hyper-parameters, but I'm not really sure that it's the case.

To load the model you can use torch.load.

state_dict = torch.load('checkpoint.pth')
print(state_dict.keys())

odict_keys(['input_to_hidden.weight', 'input_to_hidden.bias', 'hidden_1_to_hidden_2.weight', 'hidden_1_to_hidden_2.bias', 'hidden_2_to_hidden_3.weight', 'hidden_2_to_hidden_3.bias', 'hidden_3_to_output.weight', 'hidden_3_to_output.bias'])

To load the state-dict you take your instantiated but untrained model and call its load_state_dict method.

model.load_state_dict(state_dict)

Seems pretty straightforward, but as usual it's a bit more complicated. Loading the state dict works only if the model architecture is exactly the same as the checkpoint architecture. Using a model with a different architecture, this fails.

parameters = HyperParameters()
parameters.hidden_layer_1 = 400

bad_model = DropoutModel(parameters)
# This will throw an error because the tensor sizes are wrong!
bad_model.load_state_dict(state_dict)

RuntimeError: Error(s) in loading state_dict for DropoutModel:
        size mismatch for input_to_hidden.weight: copying a param of torch.Size([400, 784]) from checkpoint, where the shape is torch.Size([256, 784]) in current model.
        size mismatch for input_to_hidden.bias: copying a param of torch.Size([400]) from checkpoint, where the shape is torch.Size([256]) in current model.
        size mismatch for hidden_1_to_hidden_2.weight: copying a param of torch.Size([128, 400]) from checkpoint, where the shape is torch.Size([128, 256]) in current model.

This means we need to rebuild the model exactly as it was when trained. Information about the model architecture needs to be saved in the checkpoint, along with the state dict. To do this, you build a dictionary with all the information you need to compeletely rebuild the model.

Originally the bad-model was just called 'model' and that seems to have messed up the state-dict so I'm going to re-use the one we made before.

checkpoint = {'hyperparameters': HyperParameters,
              'state_dict': state_dict}

torch.save(checkpoint, file_name)

Remember that this is using pickle under the hood so whatever you save has to be pickleable. It probably would be safer to use parameters instead of a settings object like I did, but I didn't know we were going to be doing this.

Here's a function to load checkpoint-files.

def load_checkpoint(filepath: str) -> nn.Module:
    """Load the model checkpoint from disk

    Args:
     filepath: path to the saved checkpoint
    """
    checkpoint = torch.load(filepath)
    model = DropoutModel(checkpoint["hyperparameters"])
    model.load_state_dict(checkpoint['state_dict'])
    return model

You can see from the function that the checkpoint is really just pickling a dictionary, and we can add any arbitrary things we want to it. I'm not really sure what it gives that using pickle directly doesn't have.

model = load_checkpoint(file_name)
print(model)

DropoutModel(
  (input_to_hidden): Linear(in_features=784, out_features=256, bias=True)
  (hidden_1_to_hidden_2): Linear(in_features=256, out_features=128, bias=True)
  (hidden_2_to_hidden_3): Linear(in_features=128, out_features=64, bias=True)
  (hidden_3_to_output): Linear(in_features=64, out_features=10, bias=True)
  (dropout): Dropout(p=0.2)
)

PyTorch has more about saving and loading models in their documentation, including saving your model to continue training later (you need to save more than the model's settings).

This is from Udacity's Deep Learning Repository which supports their Deep Learning Nanodegree.

Now that you have a trained network, you can use it for making predictions. This is typically called inference, a term borrowed from statistics. However, neural networks have a tendency to perform too well on the training data and aren't able to generalize to data that hasn't been seen before. This is called overfitting and it impairs inference performance. To test for overfitting while training, we measure the performance on data not in the training set called the validation set. We avoid overfitting through regularization such as dropout while monitoring the validation performance during training.

We're going to load the dataset through torchvision but this time we'll be taking advantage of the test set which you can get by setting train=False.

The test set contains images just like the training set. Typically you'll see 10-20% of the original dataset held out for testing and validation with the rest being used for training.

We're going to use the object-oriented approach instead of the pipeline that we used earlier. It's going to have three hidden layers and one output layer.

class HyperParameters:
    inputs = 28**2
    hidden_layer_1 = 256
    hidden_layer_2 = 128
    hidden_layer_3 = 64
    outputs = 10
    axis = 1
    learning_rate = 0.003
    epochs = 30
    dropout_probability = 0.2

class Classifier(nn.Module):
    def __init__(self):
        super().__init__()
        self.input_to_hidden = nn.Linear(HyperParameters.inputs,
                                         HyperParameters.hidden_layer_1)
        self.hidden_1_to_hidden_2 = nn.Linear(HyperParameters.hidden_layer_1,
                                              HyperParameters.hidden_layer_2)
        self.hidden_2_to_hidden_3 = nn.Linear(HyperParameters.hidden_layer_2,
                                              HyperParameters.hidden_layer_3)
        self.hidden_3_to_output = nn.Linear(HyperParameters.hidden_layer_3,
                                            HyperParameters.outputs)

    def forward(self, x: torch.Tensor) -> torch.Tensor:
        """One forward-pass through the network"""
        # make sure input tensor is flattened
        x = x.view(x.shape[0], -1)

        x = F.relu(self.input_to_hidden(x))
        x = F.relu(self.hidden_1_to_hidden_2(x))
        x = F.relu(self.hidden_2_to_hidden_3(x))
        x = F.log_softmax(self.hidden_3_to_output(x),
                          dim=HyperParameters.axis)
        return x

model = Classifier()

The goal of validation is to measure the model's performance on data that isn't part of the training set. Performance here is up to the developer to define though. Typically this is just accuracy, the percentage of classes the network predicted correctly. Other options are precision and recall, top-5 error rate, etc.. We'll focus on accuracy here. First we'll do a forward pass with one batch from the test set.

Get the next image-batch.

images, labels = next(iter(test_batches))

Now we'll get the model probabilities for the image-batch.

probabilities = torch.exp(model(images))

shape = probabilities.shape
print(shape)
rows, columns = shape
assert rows == 64
assert columns == 10

torch.Size([64, 10])

With the probabilities, we can get the most likely class using the probabilities.topk method. This returns the \(k\) highest values in the tensor. Since we just want the most likely class, we can use probabilities.topk(1). This returns a tuple of the top-\(k\) values and the top-\(k\) indices. If the highest value is the fifth element, we'll get back 4 as the index.

top_p, top_class = probabilities.topk(1, dim=1)

Look at the most likely classes for the first 10 examples

print(top_class[:10,:])

tensor([[6],
        [6],
        [6],
        [6],
        [6],
        [6],
        [6],
        [6],
        [5],
        [6]])

Now we can check if the predicted classes match the labels. This is simple to do by equating top_class and labels, but we have to be careful of the shapes. Here top_class is a 2D tensor with shape (64, 1) while labels is 1D with shape (64). To get the equality to work out the way we want, top_class and labels must have the same shape.

If we do this:

equals = top_class == labels

equals will have shape (64, 64), try it yourself. What it's doing is comparing the one element in each row of top_class with each element in labels which returns 64 True/False boolean values for each row, so we have to reshape the labels first using the view method.

equals = top_class == labels.view(*top_class.shape)

Now we need to calculate the percentage of correct predictions. equals has binary values, either 0 or 1. This means that if we just sum up all the values and divide by the number of values, we get the percentage of correct predictions. This is the same operation as taking the mean, so we can get the accuracy with a call to torch.mean. If only it was that simple. If you try torch.mean(equals), you'll get an error.

RuntimeError: mean is not implemented for type torch.ByteTensor

This happens because equals has type torch.ByteTensor but torch.mean isn't implemented for tensors with that type. So we'll need to convert equals to a float tensor. Note that when we take torch.mean it returns a scalar tensor, to get the actual value as a float we'll need to do accuracy.item().

accuracy = torch.mean(equals.type(torch.FloatTensor))

print(f'Accuracy: {accuracy.item()*100}%')

Accuracy: 10.9375%

The network is untrained so it's making random guesses and we should see an accuracy around 10%. Now let's train our network and include our validation pass so we can measure how well the network is performing on the test set. Since we're not updating our parameters in the validation pass, we can speed up our code by turning off gradients using torch.no_grad():

with torch.no_grad():
    # validation pass here
    for images, labels in testloader:

Implement the validation loop below and print out the total accuracy after the loop. You can largely copy and paste the code from above, but I suggest typing it in because writing it out yourself is essential for building the skill. In general you'll always learn more by typing it rather than copy-pasting. You should be able to get an accuracy above 80%.

The train_losses and test_losses are kept for plotting later on.

def train(model, optimizer, criterion):
    train_losses, test_losses, accuracies = [], [], []
    for epoch in range(HyperParameters.epochs):
        running_loss = 0
        for images, labels in training_batches:        
            optimizer.zero_grad()
            # images = images.view(images.shape[0], -1)
            log_probabilities = model(images)
            loss = criterion(log_probabilities, labels)
            loss.backward()
            optimizer.step()
            running_loss += loss.item()        
        else:
            test_loss = 0
            accuracy = 0
            with torch.no_grad():
                for images, labels in test_batches:
                    # images = images.view(images.shape[0], -1)
                    log_probabilities = model(images)
                    test_loss += criterion(log_probabilities, labels).item()
                    probabilities = torch.exp(log_probabilities)
                    top_p, top_class = probabilities.topk(1, dim=1)
                    equals = top_class == labels.view(*top_class.shape)
                    accuracy += torch.mean(equals.type(torch.FloatTensor)).item()
            mean_accuracy = accuracy/len(test_batches)
            train_losses.append(running_loss/len(training_batches))
            test_losses.append(test_loss/len(test_batches))
            accuracies.append(mean_accuracy)
            print("Epoch: {}/{}".format(epoch + 1, HyperParameters.epochs),
                  "Training loss: {:.2f}".format(train_losses[-1]),
                  "Test Loss: {:.2f}".format(test_losses[-1]),
                  "Test Accuracy: {:.2f}".format(mean_accuracy)),
    return train_losses, test_losses, accuracies

model = Classifier()
criterion = nn.NLLLoss()
optimizer = optim.Adam(model.parameters(), lr=HyperParameters.learning_rate)

train_losses_0, test_losses_0, accuracies_0 = train(model, optimizer, criterion)

Epoch: 1/30 Training loss: 0.51 Test Loss: 0.43 Test Accuracy: 0.84
Epoch: 2/30 Training loss: 0.39 Test Loss: 0.42 Test Accuracy: 0.85
Epoch: 3/30 Training loss: 0.35 Test Loss: 0.38 Test Accuracy: 0.86
Epoch: 4/30 Training loss: 0.33 Test Loss: 0.38 Test Accuracy: 0.86
Epoch: 5/30 Training loss: 0.32 Test Loss: 0.37 Test Accuracy: 0.87
Epoch: 6/30 Training loss: 0.30 Test Loss: 0.37 Test Accuracy: 0.87
Epoch: 7/30 Training loss: 0.29 Test Loss: 0.38 Test Accuracy: 0.87
Epoch: 8/30 Training loss: 0.28 Test Loss: 0.38 Test Accuracy: 0.87
Epoch: 9/30 Training loss: 0.28 Test Loss: 0.39 Test Accuracy: 0.87
Epoch: 10/30 Training loss: 0.27 Test Loss: 0.38 Test Accuracy: 0.87
Epoch: 11/30 Training loss: 0.26 Test Loss: 0.37 Test Accuracy: 0.87
Epoch: 12/30 Training loss: 0.25 Test Loss: 0.38 Test Accuracy: 0.88
Epoch: 13/30 Training loss: 0.25 Test Loss: 0.38 Test Accuracy: 0.88
Epoch: 14/30 Training loss: 0.24 Test Loss: 0.36 Test Accuracy: 0.88
Epoch: 15/30 Training loss: 0.24 Test Loss: 0.40 Test Accuracy: 0.88
Epoch: 16/30 Training loss: 0.23 Test Loss: 0.39 Test Accuracy: 0.88
Epoch: 17/30 Training loss: 0.23 Test Loss: 0.39 Test Accuracy: 0.88
Epoch: 18/30 Training loss: 0.22 Test Loss: 0.42 Test Accuracy: 0.87
Epoch: 19/30 Training loss: 0.22 Test Loss: 0.45 Test Accuracy: 0.87
Epoch: 20/30 Training loss: 0.22 Test Loss: 0.38 Test Accuracy: 0.88
Epoch: 21/30 Training loss: 0.21 Test Loss: 0.38 Test Accuracy: 0.89
Epoch: 22/30 Training loss: 0.20 Test Loss: 0.42 Test Accuracy: 0.88
Epoch: 23/30 Training loss: 0.21 Test Loss: 0.41 Test Accuracy: 0.88
Epoch: 24/30 Training loss: 0.20 Test Loss: 0.42 Test Accuracy: 0.88
Epoch: 25/30 Training loss: 0.20 Test Loss: 0.42 Test Accuracy: 0.88
Epoch: 26/30 Training loss: 0.19 Test Loss: 0.43 Test Accuracy: 0.89
Epoch: 27/30 Training loss: 0.19 Test Loss: 0.44 Test Accuracy: 0.88
Epoch: 28/30 Training loss: 0.19 Test Loss: 0.43 Test Accuracy: 0.88
Epoch: 29/30 Training loss: 0.19 Test Loss: 0.41 Test Accuracy: 0.88
Epoch: 30/30 Training loss: 0.18 Test Loss: 0.41 Test Accuracy: 0.88

train_losses_0 = pandas.Series(train_losses_0)
accuracies_0 = pandas.Series(accuracies_0)
test_losses_0 = pandas.Series(test_losses_0n)

What do our outcomes look like?

def print_best(data: pandas.Series, label: str, decimals: int=3,
               minimum: bool=True) -> None:
    """Print a table of the best and last outcomes

    Args:
     data: the source of the information
     label: what to put in the headline
     decimals: how many decimal places to use
     minimum: whether we want the lowest score (vs the highest)
    """
    print("|{}| Value|".format(label))
    print("|-+-|")
    best = data.min() if minimum else data.max()
    best_index = data.idxmin() if minimum else data.idxmax()
    print("|Best|{{:.{}f}}|".format(decimals).format(best))
    print("|Best Location|{}|".format(best_index))
    print("|Final|{{:.{}f}}|".format(decimals).format(data.iloc[-1]))
    return

print_best(train_losses_0, "Training Loss")

So our best training loss was the final one.

print_best(test_losses_0, "Test Loss")

While the test loss was best less than halfway through the epochs.

print_best(accuracies_0, "Test Accuracy", minimum=False)

The accuracy also seems to have peaked almost at the halfway point, although the difference between the best and the final is pretty much just a rounding difference.

figure, (axe_0, axe_1) = pyplot.subplots(2, sharex=True)
figure.suptitle("Train and Test Without Dropout", weight="bold")
y_minimum = 0

# the top plot
axe_0.set_ylabel("Accuracy")

# the bottom plot
axe_1.set_xlabel("Epoch")
axe_1.set_ylabel("Loss")

test_rolling = test_losses_0.rolling(3, min_periods=1).mean()
axe_1.plot(range(HyperParameters.epochs), train_losses_0, label="Train")
axe_1.plot(range(HyperParameters.epochs), test_rolling, label="Rolling Test")
axe_1.plot(range(HyperParameters.epochs), test_losses_0, ".", alpha=0.3, label="Test")
axe_1.set_ylim(bottom=y_minimum)

axe_0.set_ylim(bottom=y_minimum)
axe_0.plot(range(len(accuracies_0)), accuracies_0, "r", label="Mean Test Accuracy")
axe_0.set_xlim((0, HyperParameters.epochs))
legend = axe.legend()

So, although the accuracy metric on the test set is pretty stable, the training loss keeps going down even as the test loss is creeping upwards. Does this imply that accuracy isn't the right metric? Log-loss differs from accuracy in that it doesn't just penalize you for what you got wrong, but also by how far you were wrong - so if you predict a high probability for the wrong label, you will get penalized more than if you predicted it but with a relatively lower probability, as opposed to accuracy which just use the binary right and wrong. So, even though our accuracy looks stable, the Log-Loss is getting worse because our model is making the same mistakes but it is getting more confident about those bad predictions. So, on to the next section where we look at one way to try and fix this.

If we look at the training and validation losses as we train the network, we can see a phenomenon known as overfitting.

The network learns the training set better and better, resulting in lower training losses. However, it starts having problems generalizing to data outside the training set leading to the validation loss increasing. The ultimate goal of any deep learning model is to make predictions on new data, so we should strive to get the lowest validation loss possible. One option is to use the version of the model with the lowest validation loss, here the one around 8-10 training epochs. This strategy is called early-stopping. In practice, you'd save the model frequently as you're training then later choose the model with the lowest validation loss.

The most common method to reduce overfitting (outside of early-stopping) is dropout, where we randomly drop input units. This forces the network to share information between weights, increasing it's ability to generalize to new data. Adding dropout in PyTorch is straightforward using the nn.Dropout module.

class Classifier(nn.Module):
    def __init__(self):
        super().__init__()
        self.fc1 = nn.Linear(784, 256)
        self.fc2 = nn.Linear(256, 128)
        self.fc3 = nn.Linear(128, 64)
        self.fc4 = nn.Linear(64, 10)

        # Dropout module with 0.2 drop probability
        self.dropout = nn.Dropout(p=0.2)

    def forward(self, x):
        # make sure input tensor is flattened
        x = x.view(x.shape[0], -1)

        # Now with dropout
        x = self.dropout(F.relu(self.fc1(x)))
        x = self.dropout(F.relu(self.fc2(x)))
        x = self.dropout(F.relu(self.fc3(x)))

        # output so no dropout here
        x = F.log_softmax(self.fc4(x), dim=1)

        return x

During training we want to use dropout to prevent overfitting, but during inference we want to use the entire network. So, we need to turn off dropout during validation, testing, and whenever we're using the network to make predictions. To do this, you use model.eval(). This sets the model to evaluation mode where the dropout probability is 0. You can turn dropout back on by setting the model to train mode with model.train(). In general, the pattern for the validation loop will look like this, where you turn off gradients, set the model to evaluation mode, calculate the validation loss and metric, then set the model back to train mode.

# Turn off gradients
with torch.no_grad():
    # set model to evaluation mode
    model.eval()

    # validation pass here
    for images, labels in testloader:
        ...

# set model back to train mode
model.train()

class DropoutModel(nn.Module):
    """Model with dropout to prevent overfitting

    Args:
     hyperparameters: object with the hyper-parameter settings
    """
    def __init__(self, hyperparameters: object=HyperParameters) -> None:
        super().__init__()
        self.input_to_hidden = nn.Linear(hyperparameters.inputs,
                                         hyperparameters.hidden_layer_1)
        self.hidden_1_to_hidden_2 = nn.Linear(hyperparameters.hidden_layer_1,
                                              hyperparameters.hidden_layer_2)
        self.hidden_2_to_hidden_3 = nn.Linear(hyperparameters.hidden_layer_2,
                                              hyperparameters.hidden_layer_3)
        self.hidden_3_to_output = nn.Linear(hyperparameters.hidden_layer_3,
                                            hyperparameters.outputs)

        # Dropout module with 0.2 drop probability
        self.dropout = nn.Dropout(p=hyperparameters.dropout_probability)
        return

    def forward(self, x: torch.Tensor) -> torch.Tensor:
        """One Forward pass through the network"""
        # make sure input tensor is flattened
        x = x.view(x.shape[0], -1)

        # Now with dropout
        x = self.dropout(F.relu(self.input_to_hidden(x)))
        x = self.dropout(F.relu(self.hidden_1_to_hidden_2(x)))
        x = self.dropout(F.relu(self.hidden_2_to_hidden_3(x)))

        # output so no dropout here
        return F.log_softmax(self.hidden_3_to_output(x),
                             dim=HyperParameters.axis)

model = DropoutModel()
criterion = nn.NLLLoss()
optimizer = optim.Adam(model.parameters(), lr=HyperParameters.learning_rate)
train_loss_1, test_loss_1, accuracies_1 = train(model, optimizer, criterion)

Epoch: 1/30 Training loss: 0.60 Test Loss: 0.53 Test Accuracy: 0.81
Epoch: 2/30 Training loss: 0.49 Test Loss: 0.49 Test Accuracy: 0.83
Epoch: 3/30 Training loss: 0.45 Test Loss: 0.47 Test Accuracy: 0.84
Epoch: 4/30 Training loss: 0.43 Test Loss: 0.48 Test Accuracy: 0.83
Epoch: 5/30 Training loss: 0.43 Test Loss: 0.47 Test Accuracy: 0.84
Epoch: 6/30 Training loss: 0.41 Test Loss: 0.45 Test Accuracy: 0.85
Epoch: 7/30 Training loss: 0.40 Test Loss: 0.45 Test Accuracy: 0.85
Epoch: 8/30 Training loss: 0.40 Test Loss: 0.49 Test Accuracy: 0.84
Epoch: 9/30 Training loss: 0.40 Test Loss: 0.47 Test Accuracy: 0.83
Epoch: 10/30 Training loss: 0.39 Test Loss: 0.44 Test Accuracy: 0.85
Epoch: 11/30 Training loss: 0.38 Test Loss: 0.46 Test Accuracy: 0.85
Epoch: 12/30 Training loss: 0.38 Test Loss: 0.49 Test Accuracy: 0.83
Epoch: 13/30 Training loss: 0.38 Test Loss: 0.44 Test Accuracy: 0.85
Epoch: 14/30 Training loss: 0.37 Test Loss: 0.43 Test Accuracy: 0.85
Epoch: 15/30 Training loss: 0.38 Test Loss: 0.46 Test Accuracy: 0.85
Epoch: 16/30 Training loss: 0.37 Test Loss: 0.47 Test Accuracy: 0.85
Epoch: 17/30 Training loss: 0.37 Test Loss: 0.46 Test Accuracy: 0.85
Epoch: 18/30 Training loss: 0.37 Test Loss: 0.54 Test Accuracy: 0.82
Epoch: 19/30 Training loss: 0.37 Test Loss: 0.44 Test Accuracy: 0.86
Epoch: 20/30 Training loss: 0.37 Test Loss: 0.45 Test Accuracy: 0.85
Epoch: 21/30 Training loss: 0.36 Test Loss: 0.45 Test Accuracy: 0.85
Epoch: 22/30 Training loss: 0.35 Test Loss: 0.47 Test Accuracy: 0.85
Epoch: 23/30 Training loss: 0.36 Test Loss: 0.45 Test Accuracy: 0.86
Epoch: 24/30 Training loss: 0.36 Test Loss: 0.46 Test Accuracy: 0.85
Epoch: 25/30 Training loss: 0.35 Test Loss: 0.46 Test Accuracy: 0.85
Epoch: 26/30 Training loss: 0.35 Test Loss: 0.48 Test Accuracy: 0.85
Epoch: 27/30 Training loss: 0.35 Test Loss: 0.46 Test Accuracy: 0.86
Epoch: 28/30 Training loss: 0.35 Test Loss: 0.45 Test Accuracy: 0.85
Epoch: 29/30 Training loss: 0.35 Test Loss: 0.47 Test Accuracy: 0.86
Epoch: 30/30 Training loss: 0.35 Test Loss: 0.46 Test Accuracy: 0.86

test_loss_1 = pandas.Series(test_loss_1)
train_loss_1 = pandas.Series(train_loss_1)
accuracies_1 = pandas.Series(accuracies_1)

def print_both(data: pandas.Series, data_2: pandas.Series, label: str,
               decimals: int=3, minimum:bool=True) -> None:
    """Prints both data sets side by side

    Args:
     data: the first data series
     data_2: the second data series
     label: something to identify the data sets
     decimals: the number of decimal places to use
     minimum: whether minimalization is the optimal
    """
    print("|{}|First|Second|".format(label))
    print("|-+-+-|")
    best = data.min() if minimum else data.max()
    best_index = data.idxmin() if minimum else data.idxmax()
    best_2 = data_2.min() if minimum else data_2.max()
    best_index_2 =  data_2.idxmin() if minimum else data_2.idxmax()
    print("|Best|{{:.{0}f}}|{{:.{0}f}}|".format(decimals).format(best, best_2))
    print("|Best Location|{}|{}|".format(best_index, best_index_2))
    print("|Final|{{:.{0}f}}|{{:.{0}f}}|".format(decimals).format(
        data.iloc[-1],
        data_2.iloc[-1]))
    return