Neural Machine Translation: Testing the Model
Table of Contents
Testing the Model
In the previous post we trained our machine translation model so now it's time to test it and see how well it does.
End
The overview post with links to all the posts in this series is here.
Raw
# # Part 4: Testing
#
# We will now be using the model you just trained to translate English sentences to German. We will implement this with two functions: The first allows you to identify the next symbol (i.e. output token). The second one takes care of combining the entire translated string.
#
# We will start by first loading in a pre-trained copy of the model you just coded. Please run the cell below to do just that.
# In[ ]:
# instantiate the model we built in eval mode
model = NMTAttn(mode='eval')
# initialize weights from a pre-trained model
model.init_from_file("model.pkl.gz", weights_only=True)
model = tl.Accelerate(model)
# <a name="4.1"></a>
# ## 4.1 Decoding
#
# As discussed in the lectures, there are several ways to get the next token when translating a sentence. For instance, we can just get the most probable token at each step (i.e. greedy decoding) or get a sample from a distribution. We can generalize the implementation of these two approaches by using the `tl.logsoftmax_sample()` method. Let's briefly look at its implementation:
#
# ```python
# def logsoftmax_sample(log_probs, temperature=1.0): # pylint: disable=invalid-name
# """Returns a sample from a log-softmax output, with temperature.
#
# Args:
# log_probs: Logarithms of probabilities (often coming from LogSofmax)
# temperature: For scaling before sampling (1.0 = default, 0.0 = pick argmax)
# """
# # This is equivalent to sampling from a softmax with temperature.
# u = np.random.uniform(low=1e-6, high=1.0 - 1e-6, size=log_probs.shape)
# g = -np.log(-np.log(u))
# return np.argmax(log_probs + g * temperature, axis=-1)
# ```
#
# The key things to take away here are: 1. it gets random samples with the same shape as your input (i.e. `log_probs`), and 2. the amount of "noise" added to the input by these random samples is scaled by a `temperature` setting. You'll notice that setting it to `0` will just make the return statement equal to getting the argmax of `log_probs`. This will come in handy later.
#
# <a name="ex06"></a>
# ### Exercise 06
#
# **Instructions:** Implement the `next_symbol()` function that takes in the `input_tokens` and the `cur_output_tokens`, then return the index of the next word. You can click below for hints in completing this exercise.
#
# <details>
# <summary>
# <font size="3" color="darkgreen"><b>Click Here for Hints</b></font>
# </summary>
# <p>
# <ul>
# <li>To get the next power of two, you can compute <i>2^log_2(token_length + 1)</i> . We add 1 to avoid <i>log(0).</i></li>
# <li>You can use <i>np.ceil()</i> to get the ceiling of a float.</li>
# <li><i>np.log2()</i> will get the logarithm base 2 of a value</li>
# <li><i>int()</i> will cast a value into an integer type</li>
# <li>From the model diagram in part 2, you know that it takes two inputs. You can feed these with this syntax to get the model outputs: <i>model((input1, input2))</i>. It's up to you to determine which variables below to substitute for input1 and input2. Remember also from the diagram that the output has two elements: [log probabilities, target tokens]. You won't need the target tokens so we assigned it to _ below for you. </li>
# <li> The log probabilities output will have the shape: (batch size, decoder length, vocab size). It will contain log probabilities for each token in the <i>cur_output_tokens</i> plus 1 for the start symbol introduced by the ShiftRight in the preattention decoder. For example, if cur_output_tokens is [1, 2, 5], the model will output an array of log probabilities each for tokens 0 (start symbol), 1, 2, and 5. To generate the next symbol, you just want to get the log probabilities associated with the last token (i.e. token 5 at index 3). You can slice the model output at [0, 3, :] to get this. It will be up to you to generalize this for any length of cur_output_tokens </li>
# </ul>
#
# In[ ]:
# UNQ_C6
# GRADED FUNCTION
def next_symbol(NMTAttn, input_tokens, cur_output_tokens, temperature):
"""Returns the index of the next token.
Args:
NMTAttn (tl.Serial): An LSTM sequence-to-sequence model with attention.
input_tokens (np.ndarray 1 x n_tokens): tokenized representation of the input sentence
cur_output_tokens (list): tokenized representation of previously translated words
temperature (float): parameter for sampling ranging from 0.0 to 1.0.
0.0: same as argmax, always pick the most probable token
1.0: sampling from the distribution (can sometimes say random things)
Returns:
int: index of the next token in the translated sentence
float: log probability of the next symbol
"""
### START CODE HERE (REPLACE INSTANCES OF `None` WITH YOUR CODE) ###
# set the length of the current output tokens
token_length = None
# calculate next power of 2 for padding length
padded_length = None
# pad cur_output_tokens up to the padded_length
padded = cur_output_tokens + None
# model expects the output to have an axis for the batch size in front so
# convert `padded` list to a numpy array with shape (x, <padded_length>) where the
# x position is the batch axis. (hint: you can use np.expand_dims() with axis=0 to insert a new axis)
padded_with_batch = None
# get the model prediction. remember to use the `NMTAttn` argument defined above.
# hint: the model accepts a tuple as input (e.g. `my_model((input1, input2))`)
output, _ = None
# get log probabilities from the last token output
log_probs = output[None]
# get the next symbol by getting a logsoftmax sample (*hint: cast to an int)
symbol = None
### END CODE HERE ###
return symbol, float(log_probs[symbol])
# In[ ]:
# BEGIN UNIT TEST
w1_unittest.test_next_symbol(next_symbol, model)
# END UNIT TEST
# Now you will implement the `sampling_decode()` function. This will call the `next_symbol()` function above several times until the next output is the end-of-sentence token (i.e. `EOS`). It takes in an input string and returns the translated version of that string.
#
# <a name="ex07"></a>
# ### Exercise 07
#
# **Instructions**: Implement the `sampling_decode()` function.
# In[ ]:
# UNQ_C7
# GRADED FUNCTION
def sampling_decode(input_sentence, NMTAttn = None, temperature=0.0, vocab_file=None, vocab_dir=None):
"""Returns the translated sentence.
Args:
input_sentence (str): sentence to translate.
NMTAttn (tl.Serial): An LSTM sequence-to-sequence model with attention.
temperature (float): parameter for sampling ranging from 0.0 to 1.0.
0.0: same as argmax, always pick the most probable token
1.0: sampling from the distribution (can sometimes say random things)
vocab_file (str): filename of the vocabulary
vocab_dir (str): path to the vocabulary file
Returns:
tuple: (list, str, float)
list of int: tokenized version of the translated sentence
float: log probability of the translated sentence
str: the translated sentence
"""
### START CODE HERE (REPLACE INSTANCES OF `None` WITH YOUR CODE) ###
# encode the input sentence
input_tokens = None
# initialize the list of output tokens
cur_output_tokens = None
# initialize an integer that represents the current output index
cur_output = None
# Set the encoding of the "end of sentence" as 1
EOS = None
# check that the current output is not the end of sentence token
while cur_output != EOS:
# update the current output token by getting the index of the next word (hint: use next_symbol)
cur_output, log_prob = None
# append the current output token to the list of output tokens
cur_output_tokens.append(cur_output)
# detokenize the output tokens
sentence = None
### END CODE HERE ###
return cur_output_tokens, log_prob, sentence
# In[ ]:
# Test the function above. Try varying the temperature setting with values from 0 to 1.
# Run it several times with each setting and see how often the output changes.
sampling_decode("I love languages.", model, temperature=0.0, vocab_file=VOCAB_FILE, vocab_dir=VOCAB_DIR)
# In[ ]:
# BEGIN UNIT TEST
w1_unittest.test_sampling_decode(sampling_decode, model)
# END UNIT TEST
# We have set a default value of `0` to the temperature setting in our implementation of `sampling_decode()` above. As you may have noticed in the `logsoftmax_sample()` method, this setting will ultimately result in greedy decoding. As mentioned in the lectures, this algorithm generates the translation by getting the most probable word at each step. It gets the argmax of the output array of your model and then returns that index. See the testing function and sample inputs below. You'll notice that the output will remain the same each time you run it.
# In[ ]:
def greedy_decode_test(sentence, NMTAttn=None, vocab_file=None, vocab_dir=None):
"""Prints the input and output of our NMTAttn model using greedy decode
Args:
sentence (str): a custom string.
NMTAttn (tl.Serial): An LSTM sequence-to-sequence model with attention.
vocab_file (str): filename of the vocabulary
vocab_dir (str): path to the vocabulary file
Returns:
str: the translated sentence
"""
_,_, translated_sentence = sampling_decode(sentence, NMTAttn, vocab_file=vocab_file, vocab_dir=vocab_dir)
print("English: ", sentence)
print("German: ", translated_sentence)
return translated_sentence
# In[ ]:
# put a custom string here
your_sentence = 'I love languages.'
greedy_decode_test(your_sentence, model, vocab_file=VOCAB_FILE, vocab_dir=VOCAB_DIR);
# In[ ]:
greedy_decode_test('You are almost done with the assignment!', model, vocab_file=VOCAB_FILE, vocab_dir=VOCAB_DIR);
# <a name="4.2"></a>
# ## 4.2 Minimum Bayes-Risk Decoding
#
# As mentioned in the lectures, getting the most probable token at each step may not necessarily produce the best results. Another approach is to do Minimum Bayes Risk Decoding or MBR. The general steps to implement this are:
#
# 1. take several random samples
# 2. score each sample against all other samples
# 3. select the one with the highest score
#
# You will be building helper functions for these steps in the following sections.
# <a name='4.2.1'></a>
# ### 4.2.1 Generating samples
#
# First, let's build a function to generate several samples. You can use the `sampling_decode()` function you developed earlier to do this easily. We want to record the token list and log probability for each sample as these will be needed in the next step.
# In[ ]:
def generate_samples(sentence, n_samples, NMTAttn=None, temperature=0.6, vocab_file=None, vocab_dir=None):
"""Generates samples using sampling_decode()
Args:
sentence (str): sentence to translate.
n_samples (int): number of samples to generate
NMTAttn (tl.Serial): An LSTM sequence-to-sequence model with attention.
temperature (float): parameter for sampling ranging from 0.0 to 1.0.
0.0: same as argmax, always pick the most probable token
1.0: sampling from the distribution (can sometimes say random things)
vocab_file (str): filename of the vocabulary
vocab_dir (str): path to the vocabulary file
Returns:
tuple: (list, list)
list of lists: token list per sample
list of floats: log probability per sample
"""
# define lists to contain samples and probabilities
samples, log_probs = [], []
# run a for loop to generate n samples
for _ in range(n_samples):
# get a sample using the sampling_decode() function
sample, logp, _ = sampling_decode(sentence, NMTAttn, temperature, vocab_file=vocab_file, vocab_dir=vocab_dir)
# append the token list to the samples list
samples.append(sample)
# append the log probability to the log_probs list
log_probs.append(logp)
return samples, log_probs
# In[ ]:
# generate 4 samples with the default temperature (0.6)
generate_samples('I love languages.', 4, model, vocab_file=VOCAB_FILE, vocab_dir=VOCAB_DIR)
# ### 4.2.2 Comparing overlaps
#
# Let us now build our functions to compare a sample against another. There are several metrics available as shown in the lectures and you can try experimenting with any one of these. For this assignment, we will be calculating scores for unigram overlaps. One of the more simple metrics is the [Jaccard similarity](https://en.wikipedia.org/wiki/Jaccard_index) which gets the intersection over union of two sets. We've already implemented it below for your perusal.
# In[ ]:
def jaccard_similarity(candidate, reference):
"""Returns the Jaccard similarity between two token lists
Args:
candidate (list of int): tokenized version of the candidate translation
reference (list of int): tokenized version of the reference translation
Returns:
float: overlap between the two token lists
"""
# convert the lists to a set to get the unique tokens
can_unigram_set, ref_unigram_set = set(candidate), set(reference)
# get the set of tokens common to both candidate and reference
joint_elems = can_unigram_set.intersection(ref_unigram_set)
# get the set of all tokens found in either candidate or reference
all_elems = can_unigram_set.union(ref_unigram_set)
# divide the number of joint elements by the number of all elements
overlap = len(joint_elems) / len(all_elems)
return overlap
# In[ ]:
# let's try using the function. remember the result here and compare with the next function below.
jaccard_similarity([1, 2, 3], [1, 2, 3, 4])
# One of the more commonly used metrics in machine translation is the ROUGE score. For unigrams, this is called ROUGE-1 and as shown in class, you can output the scores for both precision and recall when comparing two samples. To get the final score, you will want to compute the F1-score as given by:
#
# $$score = 2* \frac{(precision * recall)}{(precision + recall)}$$
#
# <a name="ex08"></a>
# ### Exercise 08
#
# **Instructions**: Implement the `rouge1_similarity()` function.
# In[ ]:
# UNQ_C8
# GRADED FUNCTION
# for making a frequency table easily
from collections import Counter
def rouge1_similarity(system, reference):
"""Returns the ROUGE-1 score between two token lists
Args:
system (list of int): tokenized version of the system translation
reference (list of int): tokenized version of the reference translation
Returns:
float: overlap between the two token lists
"""
### START CODE HERE (REPLACE INSTANCES OF `None` WITH YOUR CODE) ###
# make a frequency table of the system tokens (hint: use the Counter class)
sys_counter = None
# make a frequency table of the reference tokens (hint: use the Counter class)
ref_counter = None
# initialize overlap to 0
overlap = None
# run a for loop over the sys_counter object (can be treated as a dictionary)
for token in sys_counter:
# lookup the value of the token in the sys_counter dictionary (hint: use the get() method)
token_count_sys = None
# lookup the value of the token in the ref_counter dictionary (hint: use the get() method)
token_count_ref = None
# update the overlap by getting the smaller number between the two token counts above
overlap += None
# get the precision (i.e. number of overlapping tokens / number of system tokens)
precision = None
# get the recall (i.e. number of overlapping tokens / number of reference tokens)
recall = None
if precision + recall != 0:
# compute the f1-score
rouge1_score = None
else:
rouge1_score = 0
### END CODE HERE ###
return rouge1_score
# In[ ]:
# notice that this produces a different value from the jaccard similarity earlier
rouge1_similarity([1, 2, 3], [1, 2, 3, 4])
# In[ ]:
# BEGIN UNIT TEST
w1_unittest.test_rouge1_similarity(rouge1_similarity)
# END UNIT TEST
# ### 4.2.3 Overall score
#
# We will now build a function to generate the overall score for a particular sample. As mentioned earlier, we need to compare each sample with all other samples. For instance, if we generated 30 sentences, we will need to compare sentence 1 to sentences 2 to 30. Then, we compare sentence 2 to sentences 1 and 3 to 30, and so forth. At each step, we get the average score of all comparisons to get the overall score for a particular sample. To illustrate, these will be the steps to generate the scores of a 4-sample list.
#
# 1. Get similarity score between sample 1 and sample 2
# 2. Get similarity score between sample 1 and sample 3
# 3. Get similarity score between sample 1 and sample 4
# 4. Get average score of the first 3 steps. This will be the overall score of sample 1.
# 5. Iterate and repeat until samples 1 to 4 have overall scores.
#
# We will be storing the results in a dictionary for easy lookups.
#
# <a name="ex09"></a>
# ### Exercise 09
#
# **Instructions**: Implement the `average_overlap()` function.
# In[ ]:
# UNQ_C9
# GRADED FUNCTION
def average_overlap(similarity_fn, samples, *ignore_params):
"""Returns the arithmetic mean of each candidate sentence in the samples
Args:
similarity_fn (function): similarity function used to compute the overlap
samples (list of lists): tokenized version of the translated sentences
*ignore_params: additional parameters will be ignored
Returns:
dict: scores of each sample
key: index of the sample
value: score of the sample
"""
# initialize dictionary
scores = {}
# run a for loop for each sample
for index_candidate, candidate in enumerate(samples):
### START CODE HERE (REPLACE INSTANCES OF `None` WITH YOUR CODE) ###
# initialize overlap to 0.0
overlap = None
# run a for loop for each sample
for index_sample, sample in enumerate(samples):
# skip if the candidate index is the same as the sample index
if index_candidate == index_sample:
continue
# get the overlap between candidate and sample using the similarity function
sample_overlap = None
# add the sample overlap to the total overlap
overlap += None
# get the score for the candidate by computing the average
score = None
# save the score in the dictionary. use index as the key.
scores[index_candidate] = None
### END CODE HERE ###
return scores
# In[ ]:
average_overlap(jaccard_similarity, [[1, 2, 3], [1, 2, 4], [1, 2, 4, 5]], [0.4, 0.2, 0.5])
# In[ ]:
# BEGIN UNIT TEST
w1_unittest.test_average_overlap(average_overlap)
# END UNIT TEST
# In practice, it is also common to see the weighted mean being used to calculate the overall score instead of just the arithmetic mean. We have implemented it below and you can use it in your experiements to see which one will give better results.
# In[ ]:
def weighted_avg_overlap(similarity_fn, samples, log_probs):
"""Returns the weighted mean of each candidate sentence in the samples
Args:
samples (list of lists): tokenized version of the translated sentences
log_probs (list of float): log probability of the translated sentences
Returns:
dict: scores of each sample
key: index of the sample
value: score of the sample
"""
# initialize dictionary
scores = {}
# run a for loop for each sample
for index_candidate, candidate in enumerate(samples):
# initialize overlap and weighted sum
overlap, weight_sum = 0.0, 0.0
# run a for loop for each sample
for index_sample, (sample, logp) in enumerate(zip(samples, log_probs)):
# skip if the candidate index is the same as the sample index
if index_candidate == index_sample:
continue
# convert log probability to linear scale
sample_p = float(np.exp(logp))
# update the weighted sum
weight_sum += sample_p
# get the unigram overlap between candidate and sample
sample_overlap = similarity_fn(candidate, sample)
# update the overlap
overlap += sample_p * sample_overlap
# get the score for the candidate
score = overlap / weight_sum
# save the score in the dictionary. use index as the key.
scores[index_candidate] = score
return scores
# In[ ]:
weighted_avg_overlap(jaccard_similarity, [[1, 2, 3], [1, 2, 4], [1, 2, 4, 5]], [0.4, 0.2, 0.5])
# ### 4.2.4 Putting it all together
#
# We will now put everything together and develop the `mbr_decode()` function. Please use the helper functions you just developed to complete this. You will want to generate samples, get the score for each sample, get the highest score among all samples, then detokenize this sample to get the translated sentence.
#
# <a name="ex10"></a>
# ### Exercise 10
#
# **Instructions**: Implement the `mbr_overlap()` function.
# In[ ]:
# UNQ_C10
# GRADED FUNCTION
def mbr_decode(sentence, n_samples, score_fn, similarity_fn, NMTAttn=None, temperature=0.6, vocab_file=None, vocab_dir=None):
"""Returns the translated sentence using Minimum Bayes Risk decoding
Args:
sentence (str): sentence to translate.
n_samples (int): number of samples to generate
score_fn (function): function that generates the score for each sample
similarity_fn (function): function used to compute the overlap between a pair of samples
NMTAttn (tl.Serial): An LSTM sequence-to-sequence model with attention.
temperature (float): parameter for sampling ranging from 0.0 to 1.0.
0.0: same as argmax, always pick the most probable token
1.0: sampling from the distribution (can sometimes say random things)
vocab_file (str): filename of the vocabulary
vocab_dir (str): path to the vocabulary file
Returns:
str: the translated sentence
"""
### START CODE HERE (REPLACE INSTANCES OF `None` WITH YOUR CODE) ###
# generate samples
samples, log_probs = None
# use the scoring function to get a dictionary of scores
# pass in the relevant parameters as shown in the function definition of
# the mean methods you developed earlier
scores = None
# find the key with the highest score
max_index = None
# detokenize the token list associated with the max_index
translated_sentence = None
### END CODE HERE ###
return (translated_sentence, max_index, scores)
# In[ ]:
TEMPERATURE = 1.0
# put a custom string here
your_sentence = 'She speaks English and German.'
# In[ ]:
mbr_decode(your_sentence, 4, weighted_avg_overlap, jaccard_similarity, model, TEMPERATURE, vocab_file=VOCAB_FILE, vocab_dir=VOCAB_DIR)[0]
# In[ ]:
mbr_decode('Congratulations!', 4, average_overlap, rouge1_similarity, model, TEMPERATURE, vocab_file=VOCAB_FILE, vocab_dir=VOCAB_DIR)[0]
# In[ ]:
mbr_decode('You have completed the assignment!', 4, average_overlap, rouge1_similarity, model, TEMPERATURE, vocab_file=VOCAB_FILE, vocab_dir=VOCAB_DIR)[0]
# **This unit test take a while to run. Please be patient**
# In[ ]:
# BEGIN UNIT TEST
w1_unittest.test_mbr_decode(mbr_decode, model)
# END UNIT TEST
# #### Congratulations! Next week, you'll dive deeper into attention models and study the Transformer architecture. You will build another network but without the recurrent part. It will show that attention is all you need! It should be fun!