Bleu Score

We can express the BLEU score as:

\[ BLEU = BP\left(\prod_{i=1}^{4}precision_i\right)^{(1/4)} \]

with the Brevity Penalty and precision defined as:

\[ BP = min\left(1, e^{(1-(\textit{reference}/\textit{candidate}))}\right) \]

\[ precision_i = \frac {\sum_{snt \in{cand}}\sum_{i\in{snt}}min\Bigl(m^{i}_{cand}, m^{i}_{ref}\Bigr)}{w^{i}_{t}} \]

where:

• \(m^{i}_{cand}\), is the count of i-gram in candidate matching the reference translation.
• \(m^{i}_{ref}\), is the count of i-gram in the reference translation.
• \(w^{i}_{t}\), is the total number of i-grams in candidate translation.

And now for a meaningless plot.

data = pandas.DataFrame.from_dict({"1-gram": [0.8],
                                   "2-gram": [0.7],
                                   "3-gram": [0.6],
                                   "4-gram": [0.5]})
plot = data.hvplot.bar(title="N-Gram Precision").opts(
    width=PLOT.width,
    height=PLOT.height,
    fontscale=PLOT.fontscale,
)

output = Embed(plot=plot, file_name="n_gram_precision")()

print(output)

The n-gram precision counts how many unigrams, bigrams, trigrams, and four-grams (i=1,…,4) match their n-gram counterpart in the reference translations. This term acts as a precision metric. Unigrams account for adequacy while longer n-grams account for fluency of the translation. To avoid overcounting, the n-gram counts are clipped to the maximal n-gram count occurring in the reference (\(m_{n}^{ref}\)). Typically precision shows exponential decay with the with the degree of the n-gram.

Another meaningless plot.

data = pandas.DataFrame.from_dict({"1-gram": [0.8],
                                   "2-gram": [0.77],
                                   "3-gram": [0.74],
                                   "4-gram": [0.71]})
plot = data.hvplot.bar(title="Modified N-Gram Precision").opts(
    width=PLOT.width,
    height=PLOT.height,
    fontscale=PLOT.fontscale
)

output = Embed(plot=plot, file_name="modified_n_gram_precision")()

print(output)

When the n-gram precision is multiplied by the BP, then the exponential decay of n-grams is almost fully compensated. The BLEU score corresponds to a geometric average of this modified n-gram precision.

In this example we will have a reference translation and 2 candidates translations. We will tokenize all sentences using the NLTK.

In this section, we will show a simple pipeline for evaluating machine translated text. Due to storage and speed constraints, we will not be using our own model in this lab. Instead, we will be using Google Translate to generate English to German translations and we will evaluate it against a known evaluation set. There are three files we will need:

1. A source text in English. In this lab, we will use the first 1671 words of the wmt19 evaluation dataset downloaded via SacreBLEU. We just grabbed a subset because of limitations in the number of words that can be translated using Google Translate.
2. A reference translation to German of the corresponding first 1671 words from the original English text. This is also provided by SacreBLEU.
3. A candidate machine translation to German from the same 1671 words. This is generated by feeding the source text to a machine translation model. As mentioned above, we will use Google Translate to generate the translations in this file.

With that, we can now compare the reference an candidate translation to get the BLEU Score.

Load the raw data.

with Path(os.environ["WMT19_SOURCE"]).expanduser().open(encoding="utf-8") as reader:
    wmt19_src_1 = reader.read()

with Path(os.environ["WMT19_REFERENCE"]).expanduser().open(encoding="utf-8") as reader:
    wmt19_ref_1 = reader.read()

with Path(os.environ["WMT19_CANDIDATE"]).expanduser().open(encoding="utf-8") as reader:
    wmt19_can_1 = reader.read()

tokenized_corpus_src = nltk.word_tokenize(wmt19_src_1.lower())
tokenized_corpus_ref = nltk.word_tokenize(wmt19_ref_1.lower())
tokenized_corpus_cand = nltk.word_tokenize(wmt19_can_1.lower())    

Inspecting the first sentence of the data.

print("English source text:\n")
print(f"{wmt19_src_1[0:170]} -> {tokenized_corpus_src[0:30]}\n")
print("German reference translation:\n")
print(f"{wmt19_ref_1[0:219]} -> {tokenized_corpus_ref[0:35]}\n")
print("German machine translation:\n")
print(f"{wmt19_can_1[0:199]} -> {tokenized_corpus_cand[0:29]}")

English source text:

Welsh AMs worried about 'looking like muppets'
There is consternation among some AMs at a suggestion their title should change to MWPs (Member of the Welsh Parliament).
 -> ['\ufeffwelsh', 'ams', 'worried', 'about', "'looking", 'like', "muppets'", 'there', 'is', 'consternation', 'among', 'some', 'ams', 'at', 'a', 'suggestion', 'their', 'title', 'should', 'change', 'to', 'mwps', '(', 'member', 'of', 'the', 'welsh', 'parliament', ')', '.']

German reference translation:

Walisische Ageordnete sorgen sich "wie Dödel auszusehen"
Es herrscht Bestürzung unter einigen Mitgliedern der Versammlung über einen Vorschlag, der ihren Titel zu MWPs (Mitglied der walisischen Parlament) ändern soll.
 -> ['\ufeffwalisische', 'ageordnete', 'sorgen', 'sich', '``', 'wie', 'dödel', 'auszusehen', "''", 'es', 'herrscht', 'bestürzung', 'unter', 'einigen', 'mitgliedern', 'der', 'versammlung', 'über', 'einen', 'vorschlag', ',', 'der', 'ihren', 'titel', 'zu', 'mwps', '(', 'mitglied', 'der', 'walisischen', 'parlament', ')', 'ändern', 'soll', '.']

German machine translation:

Walisische AMs machten sich Sorgen, dass sie wie Muppets aussehen könnten
Einige AMs sind bestürzt über den Vorschlag, ihren Titel in MWPs (Mitglied des walisischen Parlaments) zu ändern.
Es ist aufg -> ['walisische', 'ams', 'machten', 'sich', 'sorgen', ',', 'dass', 'sie', 'wie', 'muppets', 'aussehen', 'könnten', 'einige', 'ams', 'sind', 'bestürzt', 'über', 'den', 'vorschlag', ',', 'ihren', 'titel', 'in', 'mwps', '(', 'mitglied', 'des', 'walisischen', 'parlaments']

print(
    "Results reference versus candidate 1 our own BLEU implementation: ",
    round(bleu_score(tokenized_corpus_cand, tokenized_corpus_ref) * 100, 1),
)

Results reference versus candidate 1 our own BLEU implementation:  43.6

print(
    "Results reference versus candidate 1 sacrebleu library BLEU: ",
    round(sacrebleu.corpus_bleu(wmt19_can_1, wmt19_ref_1).score, 1),
)

Results reference versus candidate 1 sacrebleu library BLEU:  43.2

BLEU Score Interpretation on a Corpus

From the table above (taken from here), we can see the translation is high quality (if you see "Hard to get the gist", please open your workspace, delete `wmt19_can.txt` and get the latest version via the Lab Help button). Moreover, the results of our coded BLEU score are almost identical to those of the SacreBLEU package.