Trax allows us to define neural network architectures by stacking layers (similarly to other libraries such as Keras). For this the Serial() is often used as it is a combinator that allows us to stack layers serially using function composition.

Next we'll look at a simple vanilla NN architecture containing 1 hidden(dense) layer with 128 cells and output (dense) layer with 10 cells on which we apply the final layer of LogSoftMax.

simple = layers.Serial(
  layers.Dense(128),
  layers.Relu(),
  layers.Dense(10),
  layers.LogSoftmax()
)

Each of the layers within the Serial combinator layer is considered a sublayer. Notice that unlike similar libraries, in Trax the activation functions are considered layers. To know more about the Serial layer check out the documentation for it.

Here's the representation for it.

print(simple)

Serial[
  Dense_128
  Serial[
    Relu
  ]
  Dense_10
  LogSoftmax
]

Printing the model gives you the exact same information as the model's definition itself.

By just looking at the definition you can clearly see what is going on inside the neural network. Trax is very straightforward in the way a network is defined.

To create a GRU model you will need to be familiar with the following layers (Documentation link attached with each layer name):

• ShiftRight: Shifts the tensor to the right by padding on axis 1. The mode should be specified and it refers to the context in which the model is being used. Possible values are: 'train', 'eval' or 'predict', predict mode is for fast inference. Defaults to "train".
• Embedding Maps discrete tokens to vectors. It will have shape (vocabulary length X dimension of output vectors). The dimension of output vectors (also called d_feature) is the number of elements in the word embedding.
• GRU The GRU layer. It leverages another Trax layer called GRUCell. The number of GRU units should be specified and should match the number of elements in the word embedding. If you want to stack two consecutive GRU layers, it can be done by using python's list comprehension.
• Dense Vanilla Dense layer.
• LogSoftMax Log Softmax function.

Putting everything together the GRU model looks like this.

mode = 'train'
vocab_size = 256
model_dimension = 512
n_layers = 2

GRU = layers.Serial(
      layers.ShiftRight(mode=mode),
      layers.Embedding(vocab_size=vocab_size, d_feature=model_dimension),
      [layers.GRU(n_units=model_dimension) for _ in range(n_layers)],
      layers.Dense(n_units=vocab_size),
      layers.LogSoftmax()
    )

Next is a helper function that prints information for every layer (sublayer within Serial).

Try changing the parameters defined before the GRU model and see how it changes.

def show_layers(model, layer_prefix="Serial.sublayers"):
    print(f"Total layers: {len(model.sublayers)}\n")
    for i in range(len(model.sublayers)):
        print('========')
        print(f'{layer_prefix}_{i}: {model.sublayers[i]}\n')

show_layers(GRU)

Total layers: 6

========
Serial.sublayers_0: Serial[
  ShiftRight(1)
]

========
Serial.sublayers_1: Embedding_256_512

========
Serial.sublayers_2: GRU_512

========
Serial.sublayers_3: GRU_512

========
Serial.sublayers_4: Dense_256

========
Serial.sublayers_5: LogSoftmax

print(GRU)

Serial[
  Serial[
    ShiftRight(1)
  ]
  Embedding_256_512
  GRU_512
  GRU_512
  Dense_256
  LogSoftmax
]

Interesting that it inserted a second Serial for the ShiftRight…

In this part of the notebook, we'll look at the implementation of the forward method for a vanilla RNN and implement that same method for a GRU. For this excercise we'll use a set of random weights and variables with the following dimensions:

• Embedding size (emb) : 128
• Hidden state size (h_dim) : (16,1)

The weights w_ and biases b_ are initialized with dimensions (h_dim, emb + h_dim) and (h_dim, 1). We expect the hidden state h_t to be a column vector with size (h_dim,1) and the initial hidden state h_0 is a vector of zeros.

Now we'll set up the variables for the dimensions.

Dimension = Namespace(
    embedding=128,
    hidden_variables=256,
    hidden_state=16,    
)

Now we'll initialize the various arrays.

random.seed(10)

weights = Weights(
    w1 = random.standard_normal(
        (Dimension.hidden_state,
         Dimension.embedding + Dimension.hidden_state)),
    w2 = random.standard_normal(
        (Dimension.hidden_state,
         Dimension.embedding + Dimension.hidden_state)),
    w3 = random.standard_normal(
        (Dimension.hidden_state,
         Dimension.embedding + Dimension.hidden_state)),
    b1 = random.standard_normal((Dimension.hidden_state, 1)),
    b2 = random.standard_normal((Dimension.hidden_state, 1)),
    b3 = random.standard_normal((Dimension.hidden_state, 1)),  
)

inputs = Inputs(
    hidden_state = numpy.zeros((Dimension.hidden_state, 1)),
    X = random.standard_normal((Dimension.hidden_variables, Dimension.embedding, 1))
)

The scan function is used for forward propagation in RNNs. It takes as inputs:

• fn : the function to be called recurrently (i.e. forward_GRU)
• elems : the list of inputs for each time step (X)
• weights : the parameters needed to compute fn
• h_0 : the initial hidden state

scan goes through all the elements x in elems, calls the function fn with arguments ([=x=, h_t=],=weights), stores the computed hidden state h_t and appends the result to a list ys. Complete the following cell by calling fn with arguments ([=x=, h_t=],=weights).

def scan(fn, elems, weights, h_0=None) -> tuple:
    """
    Forward propagation for RNNs

    Args:
     function: callable that updates the hidden state
      elems: input (x)
      weights: collection of weights
      h_0: the initial hidden weights
    """
    h_t = h_0
    ys = []
    for x in elems:
        y, h_t = fn([x, h_t], weights)
        ys.append(y)
    return ys, h_t

You have already seen how forward propagation is computed for vanilla RNNs and GRUs. As a quick recap, you need to have a forward method for the recurrent cell and a function like scan to go through all the elements from a sequence using a forward method. You saw that GRUs performed more computations than vanilla RNNs, and you can check that they have 3 times more parameters. In the next two cells, we compute forward propagation for a sequence with 256 time steps (T) for an RNN and a GRU with the same hidden state h_t size (=h_dim==16).

Note to future self: The default jax installation from pip is CPU only, to get it to run on the GPU (which seems to be the main reason to use it) you need to specify it. Right now the command is:

pip install jaxlib==0.1.57+cuda111 -f https://storage.googleapis.com/jax-releases/jax_releases.html

Where cuda111 refers to the fact that I have cuda 11.1 installed on the server, so I need that version. See the installation instructions for more information (and to see if anything changes).

# from python
from argparse import Namespace
from pathlib import Path

import os

# from pypi
from dotenv import load_dotenv
from trax import layers

import numpy
import trax
import trax.fastmath.numpy as trax_numpy

One important change to take into consideration is that the types of the resulting objects will be different depending on the version of numpy. With regular numpy you get numpy.ndarray but with Trax's numpy you will get jax.interpreters.xla.DeviceArray. These two types map to each other. So if you find some error logs mentioning DeviceArray type, don't worry about it, treat it like you would treat an ndarray and march ahead.

You can get a randomized numpy array by using the numpy.random.random() function.

This is one of the functionalities that Trax's numpy does not currently support in the same way as the regular numpy.

numpy_array = numpy.random.random((5,10))
print(f"The regular numpy array looks like this:\n\n {numpy_array}\n")
print(f"It is of type: {type(numpy_array)}")

The regular numpy array looks like this:

 [[0.85888927 0.37271115 0.55512878 0.95565655 0.7366696  0.81620514
  0.10108656 0.92848807 0.60910917 0.59655344]
 [0.09178413 0.34518624 0.66275252 0.44171349 0.55148779 0.70371249
  0.58940123 0.04993276 0.56179184 0.76635847]
 [0.91090833 0.09290995 0.90252139 0.46096041 0.45201847 0.99942549
  0.16242374 0.70937058 0.16062408 0.81077677]
 [0.03514717 0.53488673 0.16650012 0.30841038 0.04506241 0.23857613
  0.67483453 0.78238275 0.69520163 0.32895445]
 [0.49403187 0.52412136 0.29854125 0.46310814 0.98478429 0.50113492
  0.39807245 0.72790532 0.86333097 0.02616954]]

It is of type: <class 'numpy.ndarray'>

You can easily cast regular numpy arrays or lists into trax numpy arrays using the trax.fastmath.numpy.array() function:

trax_numpy_array = trax_numpy.array(numpy_array)
print(f"The trax numpy array looks like this:\n\n {trax_numpy_array}\n")
print(f"It is of type: {type(trax_numpy_array)}")

The trax numpy array looks like this:

 [[0.8588893  0.37271115 0.55512875 0.9556565  0.7366696  0.81620514
  0.10108656 0.9284881  0.60910916 0.59655344]
 [0.09178413 0.34518623 0.6627525  0.44171348 0.5514878  0.70371246
  0.58940125 0.04993276 0.56179184 0.7663585 ]
 [0.91090834 0.09290995 0.9025214  0.46096042 0.45201847 0.9994255
  0.16242374 0.7093706  0.16062407 0.81077677]
 [0.03514718 0.5348867  0.16650012 0.30841038 0.04506241 0.23857613
  0.67483455 0.7823827  0.69520164 0.32895446]
 [0.49403188 0.52412134 0.29854125 0.46310815 0.9847843  0.50113493
  0.39807245 0.72790533 0.86333096 0.02616954]]

It is of type: <class 'jax.interpreters.xla._DeviceArray'>

The previous section was a quick look at Trax's numpy. However this notebook also aims to teach you how you can calculate the perplexity of a trained model.

The perplexity is a metric that measures how well a probability model predicts a sample and it is commonly used to evaluate language models. It is defined as:

\[ P(W) = \sqrt[N]{\prod_{i=1}^{N} \frac{1}{P(w_i| w_1,...,w_{n-1})}} \]

As an implementation hack, you would usually take the log of that formula (to enable us to use the log probabilities we get as output of our RNN, convert exponents to products, and products into sums which makes computations less complicated and computationally more efficient). You should also take care of the padding, since you do not want to include the padding when calculating the perplexity (because we do not want to have a perplexity measure artificially good). The algebra behind this process is explained next:

We're going to use some pre-made arrays.

predictions = numpy.load(Paths.predictions)
targets = numpy.load(Paths.targets)

Now we'll cast the numpy arrays to jax.interpreters.xla.DeviceArrays.

predictions = trax_numpy.array(predictions)
targets = trax_numpy.array(targets)

print(f'predictions has shape: {predictions.shape}')
print(f'targets has shape: {targets.shape}')

predictions has shape: (32, 64, 256)
targets has shape: (32, 64)

Notice that the predictions have an extra dimension - this is the same length as the size of the vocabulary used. Because of this you will need a way of reshaping targets to match this shape. For this we will use trax.layers.one_hot.

Also note that we can get the size of the last dimension using predictions.shape[-1].

reshaped_targets = layers.one_hot(x=targets, n_categories=predictions.shape[-1])
print(f'reshaped_targets has shape: {reshaped_targets.shape}')

reshaped_targets has shape: (32, 64, 256)

By calculating the product of the predictions and the reshaped targets and summing across the last dimension, we can compute the total log perplexity.

total_log_perplexity = trax_numpy.sum(predictions * reshaped_targets, axis= -1)

Now you will need to account for the padding so this metric is not artificially deflated (since a lower perplexity means a better model). To identify which elements are padding and which are not, you can use np.equal() and get a tensor with True in the positions of actual values and False where there are paddings.

equals_zero = trax_numpy.equal(targets, 0)
print(equals_zero)

[[False False False ...  True  True  True]
 [False False False ...  True  True  True]
 [False False False ...  True  True  True]
 ...
 [False False False ...  True  True  True]
 [False False False ...  True  True  True]
 [False False False ...  True  True  True]]

equals_zero is a boolean array that has True wherever the cell had a 0 and False everywhere else. To make it numeric we can subtract the boolean array from 1 (generally in python True is treated as 1 and False as 0).

non_pad = 1.0 - equals_zero
print(f'non_pad has shape: {non_pad.shape}\n')
print(f'non_pad looks like this: \n\n {non_pad}')

non_pad has shape: (32, 64)

non_pad looks like this: 

 [[1. 1. 1. ... 0. 0. 0.]
 [1. 1. 1. ... 0. 0. 0.]
 [1. 1. 1. ... 0. 0. 0.]
 ...
 [1. 1. 1. ... 0. 0. 0.]
 [1. 1. 1. ... 0. 0. 0.]
 [1. 1. 1. ... 0. 0. 0.]]

Now if we multiply total_log_perplexity by the non_pad we'll zero-out all the entries in total_log_perplexity where non_pad has zero.

real_log_perplexity = total_log_perplexity * non_pad
print(f'real perplexity still has shape: {real_log_perplexity.shape}')

real perplexity still has shape: (32, 64)

We can check the effect of filtering out the padding by looking at the two log perplexity tensors.

print(f'log perplexity tensor before filtering padding: \n\n {total_log_perplexity}\n')
print(f'log perplexity tensor after filtering padding: \n\n {real_log_perplexity}')

log perplexity tensor before filtering padding: 

 [[ -5.396545    -1.0311184   -0.66916656 ... -22.37673    -23.18771
  -21.843483  ]
 [ -4.5857706   -1.1341286   -8.538033   ... -20.15686    -26.837097
  -23.57502   ]
 [ -5.2223887   -1.2824144   -0.17312431 ... -21.328228   -19.854412
  -33.88444   ]
 ...
 [ -5.396545   -17.291681    -4.360766   ... -20.825802   -21.065838
  -22.443115  ]
 [ -5.9313164  -14.247417    -0.2637329  ... -26.743248   -18.38433
  -22.355278  ]
 [ -5.670536    -0.10595131   0.         ... -23.332523   -28.087376
  -23.878807  ]]

log perplexity tensor after filtering padding: 

 [[ -5.396545    -1.0311184   -0.66916656 ...  -0.          -0.
   -0.        ]
 [ -4.5857706   -1.1341286   -8.538033   ...  -0.          -0.
   -0.        ]
 [ -5.2223887   -1.2824144   -0.17312431 ...  -0.          -0.
   -0.        ]
 ...
 [ -5.396545   -17.291681    -4.360766   ...  -0.          -0.
   -0.        ]
 [ -5.9313164  -14.247417    -0.2637329  ...  -0.          -0.
   -0.        ]
 [ -5.670536    -0.10595131   0.         ...  -0.          -0.
   -0.        ]]

To get a single average log perplexity across all the elements in the batch you can sum across both dimensions and divide by the number of elements. Note that the result will be the negative of the real log perplexity of the model.

log_perplexity = -trax_numpy.sum(real_log_perplexity) / trax_numpy.sum(non_pad)
print(f"log perplexity: {log_perplexity:0.4f}, "
      f"perplexity: {trax_numpy.exp(log_perplexity):0.4f}")

log perplexity: 2.3281, perplexity: 10.2586

Now that we know how to do the concatenations, horizontal and vertical, let's verify that the two formulas produce the same result.

• Formula 1: \(h^{\langle t\rangle}=g(W_{h}[h^{\langle t-1\rangle},x^{\langle t\rangle}] + b_h)\)
• Formula 2: \(h^{\langle t\rangle}=g(W_{hh}h^{\langle t-1\rangle} \oplus W_{hx}x^{\langle t\rangle} + b_h)\)

We want to assure ourselves that Formula 1 \(\Leftrightarrow\) Formula 2.

We will initially ignore the bias term \(b_h\) and the activation function g( ) because the transformation will be identical for each formula. So what we really want to compare is the result of the following parameters inside each formula:

\[ W_{h}[h^{\langle t-1\rangle},x^{\langle t\rangle}] \quad \Leftrightarrow \quad W_{hh}h^{\langle t-1\rangle} \oplus W_{hx}x^{\langle t\rangle} \]

We'll see how to do this using matrix multiplication combined with the data and techniques (stacking/concatenating) from above.

Now we'll test our model's prediction accuracy on validation data.

This program will take in a data generator and the model.

• The generator allows us to get batches of data. You can use it with a for loop:

for batch in iterator: 
   # do something with that batch

batch has dimensions (X, Y, weights).

• Column 0 corresponds to the tweet as a tensor (input).
• Column 1 corresponds to its target (actual label, positive or negative sentiment).
• Column 2 corresponds to the weights associated (example weights)
• You can feed the tweet into model and it will return the predictions for the batch.

# UNQ_C8 (UNIQUE CELL IDENTIFIER, DO NOT EDIT)
# GRADED FUNCTION: test_model
def test_model(generator: TensorGenerator, model: trax_layers.Serial) -> float:
    """Calculate the accuracy of the model

    Args: 
       generator: an iterator instance that provides batches of inputs and targets
       model: a model instance 
    Returns: 
       accuracy: float corresponding to the accuracy
    """

    accuracy = 0.
    total_num_correct = 0
    total_num_pred = 0

    ### START CODE HERE (Replace instances of 'None' with your code) ###
    for batch in generator: 

        # Retrieve the inputs from the batch
        inputs = batch[0]

        # Retrieve the targets (actual labels) from the batch
        targets = batch[1]

        # Retrieve the example weight.
        example_weight = batch[2]

        # Make predictions using the inputs
        pred = model(inputs)

        # Calculate accuracy for the batch by comparing its predictions and targets
        batch_accuracy, batch_num_correct, batch_num_pred = compute_accuracy(
            pred, targets, example_weight)

        # Update the total number of correct predictions
        # by adding the number of correct predictions from this batch
        total_num_correct += batch_num_correct

        # Update the total number of predictions 
        # by adding the number of predictions made for the batch
        total_num_pred += batch_num_pred

    # Calculate accuracy over all examples
    accuracy = total_num_correct/total_num_pred

    ### END CODE HERE ###
    return accuracy

# DO NOT EDIT THIS CELL
# testing the accuracy of your model: this takes around 20 seconds
model = trainer.training_loop.eval_model

# we used all the data for the training and validation (oops)
# so we don't have any test data. Fix that later
#accuracy = test_model(VALIDATION_GENERATOR, model)
generator = valid_generator(infinite=False)
accuracy = test_model(generator, model)
print(f'The accuracy of your model on the validation set is {accuracy:.4f}', )

The accuracy of your model on the validation set is 0.9995

Finally, let's test some custom input. You will see that deepnets are more powerful than the older methods we have used before. Although we got close to 100% accuracy using Naive Bayes and Logistic Regression, that was because the task was way easier.

This is used to predict on a new sentence.

def predict(sentence: str) -> tuple:
    """Predicts the sentiment of the sentence

    Args:
     sentence to get the sentiment for

    Returns:
     predictions, sentiment
    """
    inputs = numpy.array(converter.to_tensor(sentence))

    # Batch size 1, add dimension for batch, to work with the model
    inputs = inputs.reshape(1, len(inputs))

    # predict with the model
    probabilities = model(inputs)

    # Turn probabilities into categories
    prediction = int(probabilities[0, 1] > probabilities[0, 0])

    sentiment = "positive" if prediction == 1 else "negative"

    return prediction, sentiment

sentence = "It's such a nice day, think i'll be taking Sid to Ramsgate fish and chips for lunch at Peter's fish factory and then the beach maybe"
inputs = numpy.array(converter.to_tensor(sentence))

Deep nets allow you to understand and capture dependencies that you would have not been able to capture with a simple linear regression, or logistic regression.

• It also allows you to better use pre-trained embeddings for classification and tends to generalize better.

# from python
from functools import partial
from pathlib import Path

import random

# from pypi
from trax.supervised import training

import nltk
import trax
import trax.layers as trax_layers
import trax.fastmath.numpy as numpy

# this project
from neurotic.nlp.twitter.tensor_generator import TensorBuilder, TensorGenerator

This next part (re-downloading the dataset) is just because I have to keep setting up new containers to get trax to work…

nltk.download("twitter_samples", download_dir="/home/neurotic/data/datasets/nltk_data/")

This was defined in the last post. It seems like too much trouble not to just copy it over.

def classifier(vocab_size: int=size_of_vocabulary,
               embedding_dim: int=256,
               output_dim: int=2) -> trax_layers.Serial:
    """Creates the classifier model

    Args:
     vocab_size: number of tokens in the training vocabulary
     embedding_dim: output dimension for the Embedding layer
     output_dim: dimension for the Dense layer

    Returns:
     the composed layer-model
    """
    embed_layer = trax_layers.Embedding(
        vocab_size=vocab_size, # Size of the vocabulary
        d_feature=embedding_dim)  # Embedding dimension

    mean_layer = trax_layers.Mean(axis=1)

    dense_output_layer = trax_layers.Dense(n_units = output_dim)

    log_softmax_layer = trax_layers.LogSoftmax()

    model = trax_layers.Serial(
      embed_layer,
      mean_layer,
      dense_output_layer,
      log_softmax_layer
    )
    return model

Now to train the model.

First define the TrainTask, EvalTask and Loop in preparation to training the model.

random.seed(271)

# train_generator(batch_size=batch_size, shuffle=True),

train_task = training.TrainTask(
    labeled_data=training_generator,
    loss_layer=trax_layers.CrossEntropyLoss(),
    optimizer=trax.optimizers.Adam(0.01),
    n_steps_per_checkpoint=10,
)

eval_task = training.EvalTask(
    labeled_data=validation_generator,
    metrics=[trax_layers.CrossEntropyLoss(), trax_layers.Accuracy()],
)

model = classifier()

This defines a model trained using tl.CrossEntropyLoss optimized with the trax.optimizers.Adam optimizer, all the while tracking the accuracy using tl.Accuracy metric. We also track tl.CrossEntropyLoss on the validation set.

Now let's make an output directory and train the model.

output_path = Path("~/models/").expanduser()
if not output_path.is_dir():
    output_path.mkdir()

def train_model(classifier, train_task, eval_task, n_steps, output_dir):
    """Create and run the training loop

    Args: 
       classifier - the model you are building
       train_task - Training task
       eval_task - Evaluation task
       n_steps - the evaluation steps
       output_dir - folder to save your files
    Returns:
       trainer -  trax trainer
    """
    training_loop = training.Loop(
                                model=classifier, # The learning model
                                tasks=train_task, # The training task
                                eval_tasks = eval_task, # The evaluation task
                                output_dir = output_dir) # The output directory

    training_loop.run(n_steps = n_steps)
    # Return the training_loop, since it has the model.
    return training_loop

training_loop = train_model(model, train_task, eval_task, 100, output_path)

Step    110: Ran 10 train steps in 6.06 secs
Step    110: train CrossEntropyLoss |  0.00527583
Step    110: eval  CrossEntropyLoss |  0.00304692
Step    110: eval          Accuracy |  1.00000000

Step    120: Ran 10 train steps in 2.06 secs
Step    120: train CrossEntropyLoss |  0.02130376
Step    120: eval  CrossEntropyLoss |  0.00000677
Step    120: eval          Accuracy |  1.00000000

Step    130: Ran 10 train steps in 0.75 secs
Step    130: train CrossEntropyLoss |  0.01026674
Step    130: eval  CrossEntropyLoss |  0.00424393
Step    130: eval          Accuracy |  1.00000000

Step    140: Ran 10 train steps in 1.33 secs
Step    140: train CrossEntropyLoss |  0.00172522
Step    140: eval  CrossEntropyLoss |  0.00004072
Step    140: eval          Accuracy |  1.00000000

Step    150: Ran 10 train steps in 0.77 secs
Step    150: train CrossEntropyLoss |  0.00002847
Step    150: eval  CrossEntropyLoss |  0.00000232
Step    150: eval          Accuracy |  1.00000000

Step    160: Ran 10 train steps in 0.78 secs
Step    160: train CrossEntropyLoss |  0.00002123
Step    160: eval  CrossEntropyLoss |  0.00104654
Step    160: eval          Accuracy |  1.00000000

Step    170: Ran 10 train steps in 0.79 secs
Step    170: train CrossEntropyLoss |  0.00001706
Step    170: eval  CrossEntropyLoss |  0.00000080
Step    170: eval          Accuracy |  1.00000000

Step    180: Ran 10 train steps in 0.83 secs
Step    180: train CrossEntropyLoss |  0.00001554
Step    180: eval  CrossEntropyLoss |  0.00000989
Step    180: eval          Accuracy |  1.00000000

Step    190: Ran 10 train steps in 0.85 secs
Step    190: train CrossEntropyLoss |  0.00639312
Step    190: eval  CrossEntropyLoss |  0.00255337
Step    190: eval          Accuracy |  1.00000000

Step    200: Ran 10 train steps in 0.85 secs
Step    200: train CrossEntropyLoss |  0.00124322
Step    200: eval  CrossEntropyLoss |  0.02190475
Step    200: eval          Accuracy |  1.00000000

Now that you have trained a model, you can access it as training_loop.model object. We will actually use training_loop.eval_model and in the next weeks you will learn why we sometimes use a different model for evaluation, e.g., one without dropout. For now, make predictions with your model.

Use the training data just to see how the prediction process works.

• Later, you will use validation data to evaluate your model's performance.

Create a generator object.

tmp_train_generator = train_generator(batch_size=16)

Get one batch.

tmp_batch = next(tmp_train_generator)

Position 0 has the model inputs (tweets as tensors). Position 1 has the targets (the actual labels).

tmp_inputs, tmp_targets, tmp_example_weights = tmp_batch

print(f"The batch is a tuple of length {len(tmp_batch)} because position 0 contains the tweets, and position 1 contains the targets.") 
print(f"The shape of the tweet tensors is {tmp_inputs.shape} (num of examples, length of tweet tensors)")
print(f"The shape of the labels is {tmp_targets.shape}, which is the batch size.")
print(f"The shape of the example_weights is {tmp_example_weights.shape}, which is the same as inputs/targets size.")

The batch is a tuple of length 3 because position 0 contains the tweets, and position 1 contains the targets.
The shape of the tweet tensors is (16, 14) (num of examples, length of tweet tensors)
The shape of the labels is (16,), which is the batch size.
The shape of the example_weights is (16,), which is the same as inputs/targets size.

Feed the tweet tensors into the model to get a prediction.

tmp_pred = training_loop.eval_model(tmp_inputs)
print(f"The prediction shape is {tmp_pred.shape}, num of tensor_tweets as rows")
print("Column 0 is the probability of a negative sentiment (class 0)")
print("Column 1 is the probability of a positive sentiment (class 1)")
print()
print("View the prediction array")
print(tmp_pred)

The prediction shape is (16, 2), num of tensor_tweets as rows
Column 0 is the probability of a negative sentiment (class 0)
Column 1 is the probability of a positive sentiment (class 1)

View the prediction array
[[-1.2960873e+01 -2.3841858e-06]
 [-5.6474457e+00 -3.5326481e-03]
 [-5.3460855e+00 -4.7781467e-03]
 [-7.6736917e+00 -4.6515465e-04]
 [-5.2682662e+00 -5.1658154e-03]
 [-1.0566207e+01 -2.5749207e-05]
 [-5.6388092e+00 -3.5634041e-03]
 [-3.9540453e+00 -1.9363165e-02]
 [ 0.0000000e+00 -2.0700916e+01]
 [ 0.0000000e+00 -2.2949795e+01]
 [ 0.0000000e+00 -2.3168846e+01]
 [ 0.0000000e+00 -2.4553205e+01]
 [-9.5367432e-07 -1.3878939e+01]
 [ 0.0000000e+00 -1.6655178e+01]
 [ 0.0000000e+00 -1.5975946e+01]
 [ 0.0000000e+00 -2.0577690e+01]]

To turn these probabilities into categories (negative or positive sentiment prediction), for each row:

• Compare the probabilities in each column.
• If column 1 has a value greater than column 0, classify that as a positive tweet.
• Otherwise if column 1 is less than or equal to column 0, classify that example as a negative tweet.

Turn probabilites into category predictions.

tmp_is_positive = tmp_pred[:,1] > tmp_pred[:,0]
for i, p in enumerate(tmp_is_positive):
    print(f"Neg log prob {tmp_pred[i,0]:.4f}\tPos log prob {tmp_pred[i,1]:.4f}\t is positive? {p}\t actual {tmp_targets[i]}")

Neg log prob -12.9609   Pos log prob -0.0000     is positive? True       actual 1
Neg log prob -5.6474    Pos log prob -0.0035     is positive? True       actual 1
Neg log prob -5.3461    Pos log prob -0.0048     is positive? True       actual 1
Neg log prob -7.6737    Pos log prob -0.0005     is positive? True       actual 1
Neg log prob -5.2683    Pos log prob -0.0052     is positive? True       actual 1
Neg log prob -10.5662   Pos log prob -0.0000     is positive? True       actual 1
Neg log prob -5.6388    Pos log prob -0.0036     is positive? True       actual 1
Neg log prob -3.9540    Pos log prob -0.0194     is positive? True       actual 1
Neg log prob 0.0000     Pos log prob -20.7009    is positive? False      actual 0
Neg log prob 0.0000     Pos log prob -22.9498    is positive? False      actual 0
Neg log prob 0.0000     Pos log prob -23.1688    is positive? False      actual 0
Neg log prob 0.0000     Pos log prob -24.5532    is positive? False      actual 0
Neg log prob -0.0000    Pos log prob -13.8789    is positive? False      actual 0
Neg log prob 0.0000     Pos log prob -16.6552    is positive? False      actual 0
Neg log prob 0.0000     Pos log prob -15.9759    is positive? False      actual 0
Neg log prob 0.0000     Pos log prob -20.5777    is positive? False      actual 0

Notice that since you are making a prediction using a training batch, it's more likely that the model's predictions match the actual targets (labels).

• Every prediction that the tweet is positive is also matching the actual target of 1 (positive sentiment).
• Similarly, all predictions that the sentiment is not positive matches the actual target of 0 (negative sentiment)

One more useful thing to know is how to compare if the prediction is matching the actual target (label).

• The result of calculation is_positive is a boolean.
• The target is a type trax.fastmath.numpy.int32
• If you expect to be doing division, you may prefer to work with decimal numbers with the data type type trax.fastmath.numpy.int32

View the array of booleans.

print("Array of booleans")
display(tmp_is_positive)

Array of booleans
DeviceArray([ True,  True,  True,  True,  True,  True,  True,  True,
             False, False, False, False, False, False, False, False],            dtype=bool)

Convert booleans to type int32.

• True is converted to 1
• False is converted to 0

tmp_is_positive_int = tmp_is_positive.astype(trax.fastmath.numpy.int32)

View the array of integers.

print("Array of integers")
display(tmp_is_positive_int)

Array of integers
DeviceArray([1, 1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0], dtype=int32)

Convert boolean to type float32.

tmp_is_positive_float = tmp_is_positive.astype(numpy.float32)

View the array of floats.

print("Array of floats")
display(tmp_is_positive_float)

Array of floats
DeviceArray([1., 1., 1., 1., 1., 1., 1., 1., 0., 0., 0., 0., 0., 0., 0.,
             0.], dtype=float32)

print(tmp_pred.shape)

(16, 2)

Note that Python usually does type conversion for you when you compare a boolean to an integer.

• True compared to 1 is True, otherwise any other integer is False.
• False compared to 0 is True, otherwise any ohter integer is False.

print(f"True == 1: {True == 1}")
print(f"True == 2: {True == 2}")
print(f"False == 0: {False == 0}")
print(f"False == 2: {False == 2}")

True == 1: True
True == 2: False
False == 0: True
False == 2: False

However, we recommend that you keep track of the data type of your variables to avoid unexpected outcomes. So it helps to convert the booleans into integers.

Some aliases to get closer to what the notebook has.

numpy_fastmath = fastmath.numpy
random = fastmath.random

This will be the base class that the others will inherit from.

@attr.s(auto_attribs=True)
class Layer:
    """Base class for layers
    """
    def forward(self, x: numpy.ndarray):
        """The forward propagation method

       Raises:
        NotImplementedError - method is called but child hasn't implemented it
       """
        raise NotImplementedError

    def init_weights_and_state(self, input_signature, random_key):
        """method to initialize the weights
       based on the input signature and random key,
       be implemented by subclasses of this Layer class
       """
        raise NotImplementedError

    def init(self, input_signature, random_key) -> numpy.ndarray:
        """initializes and returns the weights

       Note:
        This is just an alias for the ``init_weights_and_state``
       method for some reason

       Args: 
        input_signature: who knows?
        random_key: once again, who knows?

       Returns:
        the weights
       """
        self.init_weights_and_state(input_signature, random_key)
        return self.weights

    def __call__(self, x) -> numpy.ndarray:
        """This is an alias for the ``forward`` method

       Args:
        x: input array

       Returns:
        whatever the ``forward`` method does
       """
        return self.forward(x)

Here's the ReLU function:

\[ \mathrm{ReLU}(x) = \mathrm{max}(0,x) \]

We'll implement the ReLU activation function below. The function will take in a matrix or vector and it transform all the negative numbers into 0 while keeping all the positive numbers intact.

Please use numpy.maximum(A,k) to find the maximum between each element in A and a scalar k.

class Relu(Layer):
    """Relu activation function implementation"""
    def forward(self, x: numpy.ndarray) -> numpy.ndarray:
        """"Performs the activation

       Args: 
           - x: the input

       Returns:
           - activation: all positive or 0 version of x
       """
        return numpy.maximum(x, 0)

Implement the forward function of the Dense class.

• The forward function multiplies the input to the layer (x) by the weight matrix (W).

\[ \mathrm{forward}(\mathbf{x},\mathbf{W}) = \mathbf{xW} \]

• You can use numpy.dot to perform the matrix multiplication.

Note that for more efficient code execution, you will use the trax version of math, which includes a trax version of numpy and also random.

Implement the weight initializer new_weights function

• Weights are initialized with a random key.
• The second parameter is a tuple for the desired shape of the weights (num_rows, num_cols)
• The num of rows for weights should equal the number of columns in x, because for forward propagation, you will multiply x times weights.

Please use trax.fastmath.random.normal(key, shape, dtype=tf.float32) to generate random values for the weight matrix. The key difference between this function and the standard numpy randomness is the explicit use of random keys, which need to be passed in. While it can look tedious at the first sight to pass the random key everywhere, you will learn in Course 4 why this is very helpful when implementing some advanced models.

• key can be generated by calling random.get_prng(seed) and passing in a number for the seed.
• shape is a tuple with the desired shape of the weight matrix.
  • The number of rows in the weight matrix should equal the number of columns in the variable x. Since x may have 2 dimensions if it represents a single training example (row, col), or three dimensions (batch_size, row, col), get the last dimension from the tuple that holds the dimensions of x.
  • The number of columns in the weight matrix is the number of units chosen for that dense layer. Look at the __init__ function to see which variable stores the number of units.
• dtype is the data type of the values in the generated matrix; keep the default of tf.float32. In this case, don't explicitly set the dtype (just let it use the default value).

Set the standard deviation of the random values to 0.1

• The values generated have a mean of 0 and standard deviation of 1.
• Set the default standard deviation stdev to be 0.1 by multiplying the standard deviation to each of the values in the weight matrix.

See how the fastmath.trax.random.normal function works.

tmp_key = random.get_prng(seed=1)
print("The random seed generated by random.get_prng")
display(tmp_key)

WARNING:absl:No GPU/TPU found, falling back to CPU. (Set TF_CPP_MIN_LOG_LEVEL=0 and rerun for more info.)
The random seed generated by random.get_prng
DeviceArray([0, 1], dtype=uint32)

For some reason tensorflow can't find the GPU. Setting the log level to 0 like the message suggests shows that it gives up after trying to find a TPU, there's no indication that it's looking for the GPU.

import tensorflow
print(tensorflow.test.gpu_device_name())

Hmmm. I'll have to troubleshoot that.

print("choose a matrix with 2 rows and 3 columns")
tmp_shape=(2,3)
print(tmp_shape)

choose a matrix with 2 rows and 3 columns
(2, 3)

Generate a weight matrix Note that you'll get an error if you try to set dtype to tf.float32, where tf is tensorflow Just avoid setting the dtype and allow it to use the default data type

tmp_weight = random.normal(key=tmp_key, shape=tmp_shape)

print("Weight matrix generated with a normal distribution with mean 0 and stdev of 1")
display(tmp_weight)

Weight matrix generated with a normal distribution with mean 0 and stdev of 1
DeviceArray([[ 0.957307  , -0.9699291 ,  1.0070664 ],
             [ 0.36619022,  0.17294823,  0.29092228]], dtype=float32)

@attr.s(auto_attribs=True)
class Dense(Layer):
    """
    A dense (fully-connected) layer.

    Args:
     - n_units: the number of columns for our weight matrix
     - init_stdev: standard deviation for our initial weights
    """
    n_units: int
    init_stdev: float=0.1

    def forward(self, x: numpy.ndarray) -> numpy.ndarray:
        """The dot product of the input and the weights

       Args:
        x: input to multipyl

       Returns:
        product of x and weights
       """
        return numpy.dot(x, self.weights)

    def init_weights_and_state(self, input_signature: tuple,
                               random_key: int) -> numpy.ndarray:
        """initializes the weights

       Args:
        input_signature: tuple whose final dimension will be the number of rows
        random_ke: something to start the random normal generator with
       """
        input_shape = input_signature.shape

        # to allow for more than two-dimensional matrices,
        # we use the last column of the input shape, rather than assuming it's
        # column 1
        self.weights = (random.normal(key=random_key,
                                      shape=(input_shape[-1], self.n_units))
             * self.init_stdev)
        return self.weights

dense_layer = Dense(n_units=10)  #sets  number of units in dense layer
random_key = random.get_prng(seed=0)  # sets random seed
z = numpy.array([[2.0, 7.0, 25.0]]) # input array 

dense_layer.init(z, random_key)
print("Weights are\n ",dense_layer.weights) #Returns randomly generated weights
output = dense_layer(z)
print("Foward function output is ", output) # Returns multiplied values of units and weights

expected_weights = numpy.array([
    [-0.02837108,  0.09368162, -0.10050076,  0.14165013,  0.10543301,  0.09108126,
     -0.04265672,  0.0986188,  -0.05575325,  0.00153249],
    [-0.20785688,  0.0554837,   0.09142365,  0.05744595,  0.07227863,  0.01210617,
     -0.03237354,  0.16234995,  0.02450038, -0.13809784],
    [-0.06111237,  0.01403724,  0.08410042, -0.1094358,  -0.10775021, -0.11396459,
     -0.05933381, -0.01557652, -0.03832145, -0.11144515]])

expected_output = numpy.array(
    [[-3.0395496,   0.9266802,   2.5414743,  -2.050473,   -1.9769388,  -2.582209,
      -1.7952735,   0.94427425, -0.8980402,  -3.7497487]])

expect(numpy.allclose(dense_layer.weights, expected_weights)).to(be_true)
expect(numpy.allclose(output, expected_output)).to(be_true)

Weights are
  [[-0.02837108  0.09368162 -0.10050076  0.14165013  0.10543301  0.09108126
  -0.04265672  0.0986188  -0.05575325  0.00153249]
 [-0.20785688  0.0554837   0.09142365  0.05744595  0.07227863  0.01210617
  -0.03237354  0.16234995  0.02450038 -0.13809784]
 [-0.06111237  0.01403724  0.08410042 -0.1094358  -0.10775021 -0.11396459
  -0.05933381 -0.01557652 -0.03832145 -0.11144515]]
Foward function output is  [[-3.03954965  0.92668021  2.54147445 -2.05047299 -1.97693891 -2.58220917
  -1.79527355  0.94427423 -0.89804017 -3.74974866]]

For the model implementation we will use the Trax layers library. Trax layers are very similar to the ones we implemented above, but in addition to trainable weights they also have a non-trainable state. This state is used in layers like batch normalization and for inference - we will learn more about it later on.

First, look at the code of the Trax Dense layer and compare to the implementation above.

• Trax Dense layer implementation

Another other important layer that we will use a lot is the Serial layer which allows us to execute one layer after another in sequence.

• You can pass in the layers as arguments to Serial, separated by commas.
• For example: tl.Serial(tl.Embeddings(...), tl.Mean(...), tl.Dense(...), tl.LogSoftmax(...))

The layer classes have pretty good docstrings, unlike the fastmath stuff, so it might be useful to look at it - but it's too long to include here.

We're also going to use an Embedding

• tl.Embedding(vocab_size, d_feature).
• vocab_size is the number of unique words in the given vocabulary.
• d_feature is the number of elements in the word embedding (some choices for a word embedding size range from 150 to 300, for example).

tmp_embed = trax_layers.Embedding(vocab_size=3, d_feature=2)
display(tmp_embed)

Embedding_3_2

Another useful layer is the Mean which calculates means across an axis. In this case, use axis = 1 (across rows) to get an average embedding vector (an embedding vector that is an average of all words in the vocabulary).

• For example, if the embedding matrix is 300 elements and vocab size is 10,000 words, taking the mean of the embedding matrix along axis=1 will yield a vector of 300 elements.

Pretend the embedding matrix uses 2 elements for embedding the meaning of a word and has a vocabulary size of 3, so it has shape (2,3).

tmp_embed = numpy.array([[1,2,3,],
                         [4,5,6]
                         ])

First take the mean along axis 0, which creates a vector whose length equals the vocabulary size (the number of columns).

display(numpy.mean(tmp_embed,axis=0))

array([2.5, 3.5, 4.5])

If you take the mean along axis 1 it creates a vector whose length equals the number of elements in a word embedding (the rows).

display(numpy.mean(tmp_embed,axis=1))

array([2., 5.])

Finally, a LogSoftmax layer gives you a log-softmax output.

builder = TensorBuilder()
size_of_vocabulary = len(builder.vocabulary)

def classifier(vocab_size: int=size_of_vocabulary,
               embedding_dim: int=256,
               output_dim: int=2) -> trax_layers.Serial:
    """Creates the classifier model

    Args:
     vocab_size: number of tokens in the training vocabulary
     embedding_dim: output dimension for the Embedding layer
     output_dim: dimension for the Dense layer

    Returns:
     the composed layer-model
    """
    embed_layer = trax_layers.Embedding(
        vocab_size=vocab_size, # Size of the vocabulary
        d_feature=embedding_dim)  # Embedding dimension

    mean_layer = trax_layers.Mean(axis=1)

    dense_output_layer = trax_layers.Dense(n_units = output_dim)

    log_softmax_layer = trax_layers.LogSoftmax()

    model = trax_layers.Serial(
      embed_layer,
      mean_layer,
      dense_output_layer,
      log_softmax_layer
    )
    return model

tmp_model = classifier()

print(type(tmp_model))
display(tmp_model)

<class 'trax.layers.combinators.Serial'>
Serial[
  Embedding_9164_256
  Mean
  Dense_2
  LogSoftmax
]

The NLTK data has to be downloaded at least once.

nltk.download("twitter_samples", download_dir="~/data/datasets/nltk_data/")

positive = twitter_samples.strings('positive_tweets.json')
negative = twitter_samples.strings('negative_tweets.json')

print(f"Positive Tweets: {len(positive):,}")
print(f"Negative Tweets: {len(negative):,}")

Positive Tweets: 5,000
Negative Tweets: 5,000

Instead of randomly splitting the data we're going to do a straight slice.

SPLIT = 4000

Now build the vocabulary.

• Map each word in each tweet to an integer (an "index").
• The following code does this for you, but please read it and understand what it's doing.
• Note that you will build the vocabulary based on the training data.
• To do so, you will assign an index to everyword by iterating over your training set.

The vocabulary will also include some special tokens

• __PAD__: padding
• </e>: end of line
• __UNK__: a token representing any word that is not in the vocabulary.

Tokens = Namespace(padding="__PAD__", ending="__</e>__", unknown="__UNK__")
process = TwitterProcessor()
vocabulary = {Tokens.padding: 0, Tokens.ending: 1, Tokens.unknown: 2}
for tweet in train_x:
    for token in process(tweet):
        if token not in vocabulary:
            vocabulary[token] = len(vocabulary)

print(f"Words in the vocabulary: {len(vocabulary):,}")

count = 0
for token in vocabulary:
    print(f"{count}: {token}: {vocabulary[token]}")
    count += 1
    if count == 5:
        break

Words in the vocabulary: 9,164
0: __PAD__: 0
1: __</e>__: 1
2: __UNK__: 2
3: followfriday: 3
4: top: 4

Now we'll write a function that will convert each tweet to a tensor (a list of unique integer IDs representing the processed tweet).

• Note, the returned data type will be a regular Python `list()`
  • You won't use TensorFlow in this function
  • You also won't use a numpy array
  • You also won't use trax.fastmath.numpy array

• For words in the tweet that are not in the vocabulary, set them to the unique ID for the token `__UNK__`.

  For example, given this string:

'@happypuppy, is Maria happy?'

You first tokenize it.

['maria', 'happi']

Then convert each word into the index for it.

[2, 56]

Notice that the word "maria" is not in the vocabulary, so it is assigned the unique integer associated with the __UNK__ token, because it is considered "unknown."

def tweet_to_tensor(tweet: str, vocab_dict: dict,
                    unk_token: str='__UNK__', verbose: bool=False):
    """Convert a tweet to a list of indices

    Args: 
       tweet - A string containing a tweet
       vocab_dict - The words dictionary
       unk_token - The special string for unknown tokens
       verbose - Print info during runtime

    Returns:
       tensor_l - A python list with indices for the tweet tokens
    """
    # Process the tweet into a list of words
    # where only important words are kept (stop words removed)
    word_l = processor(tweet)

    if verbose:
        print("List of words from the processed tweet:")
        print(word_l)

    # Initialize the list that will contain the unique integer IDs of each word
    tensor_l = []

    # Get the unique integer ID of the __UNK__ token
    unk_ID = vocab_dict[unk_token]

    if verbose:
        print(f"The unique integer ID for the unk_token is {unk_ID}")

    # for each word in the list:
    for word in word_l:

        # Get the unique integer ID.
        # If the word doesn't exist in the vocab dictionary,
        # use the unique ID for __UNK__ instead.
        word_ID = vocab_dict.get(word, unk_ID)

        # Append the unique integer ID to the tensor list.
        tensor_l.append(word_ID) 

    return tensor_l

print("Actual tweet is\n", positive_validation[0])
print("\nTensor of tweet:\n", tweet_to_tensor(positive_validation[0], vocab_dict=vocabulary))

Actual tweet is
 Bro:U wan cut hair anot,ur hair long Liao bo
Me:since ord liao,take it easy lor treat as save $ leave it longer :)
Bro:LOL Sibei xialan

Tensor of tweet:
 [1072, 96, 484, 2376, 750, 8220, 1132, 750, 53, 2, 2701, 796, 2, 2, 354, 606, 2, 3523, 1025, 602, 4599, 9, 1072, 158, 2, 2]

def test_tweet_to_tensor():
    test_cases = [

        {
            "name":"simple_test_check",
            "input": [positive_validation[1], vocabulary],
            "expected":[444, 2, 304, 567, 56, 9],
            "error":"The function gives bad output for val_pos[1]. Test failed"
        },
        {
            "name":"datatype_check",
            "input":[positive_validation[1], vocabulary],
            "expected":type([]),
            "error":"Datatype mismatch. Need only list not np.array"
        },
        {
            "name":"without_unk_check",
            "input":[positive_validation[1], vocabulary],
            "expected":6,
            "error":"Unk word check not done- Please check if you included mapping for unknown word"
        }
    ]
    count = 0
    for test_case in test_cases:        
        try:
            if test_case['name'] == "simple_test_check":
                assert test_case["expected"] == tweet_to_tensor(*test_case['input'])
                count += 1
            if test_case['name'] == "datatype_check":
                assert isinstance(tweet_to_tensor(*test_case['input']), test_case["expected"])
                count += 1
            if test_case['name'] == "without_unk_check":
                assert None not in tweet_to_tensor(*test_case['input'])
                count += 1

        except:
            print(test_case['error'])
    if count == 3:
        print("\033[92m All tests passed")
    else:
        print(count," Tests passed out of 3")
test_tweet_to_tensor()            

The function gives bad output for val_pos[1]. Test failed
2  Tests passed out of 3

Their tweet processor wipes out everything after the start of a URL, even if it isn't part of the URL, so they have fewer tokens, so the indices won't match exactly.

Most of the time in Natural Language Processing, and AI in general we use batches when training our data sets.

• If instead of training with batches of examples, you were to train a model with one example at a time, it would take a very long time to train the model.
• You will now build a data generator that takes in the positive/negative tweets and returns a batch of training examples. It returns the model inputs, the targets (positive or negative labels) and the weight for each target (ex: this allows us to treat some examples as more important to get right than others, but commonly this will all be 1.0).

Once you create the generator, you could include it in a for loop:

for batch_inputs, batch_targets, batch_example_weights in data_generator:

You can also get a single batch like this:

batch_inputs, batch_targets, batch_example_weights = next(data_generator)

The generator returns the next batch each time it's called.

• This generator returns the data in a format (tensors) that you could directly use in your model.
• It returns a triple: the inputs, targets, and loss weights:

– Inputs is a tensor that contains the batch of tweets we put into the model. – Targets is the corresponding batch of labels that we train to generate. – Loss weights here are just 1s with same shape as targets. Next week, you will use it to mask input padding.