Download this tutorial

Google Neural Machine Translation (GNMT)

In this notebook, we are going to train Google NMT on IWSLT 2015 English-Vietnamese Dataset. The building process includes four key steps:

  1. Load and preprocess the dataset
  2. Create a sampler and DataLoader
  3. Build the actual model
  4. Write the training algorithm

This tutorial will guide you through each of the steps and explain briefly how each works. Please remember to click the download button at the top of the page to download the necessary files to follow this tutorial.

Setup

Firstly, we need to setup the environment and import the necessary modules. For this tutorial, a GPU is highly important.

In [1]:
import warnings
warnings.filterwarnings('ignore')

import argparse
import time
import random
import os
import io
import logging
import numpy as np
import mxnet as mx
from mxnet import gluon
import gluonnlp as nlp
import nmt

Next, we need to specify the hyperparameters for the dataset, the model, and for training and testing time.

In [2]:
np.random.seed(100)
random.seed(100)
mx.random.seed(10000)
ctx = mx.gpu(0)

# parameters for dataset
dataset = 'IWSLT2015'
src_lang, tgt_lang = 'en', 'vi'
src_max_len, tgt_max_len = 50, 50

# parameters for model
num_hidden = 512
num_layers = 2
num_bi_layers = 1
dropout = 0.2

# parameters for training
batch_size, test_batch_size = 128, 32
num_buckets = 5
epochs = 1
clip = 5
lr = 0.001
lr_update_factor = 0.5
log_interval = 10
save_dir = 'gnmt_en_vi_u512'

#parameters for testing
beam_size = 10
lp_alpha = 1.0
lp_k = 5

nmt.utils.logging_config(save_dir)
All Logs will be saved to gnmt_en_vi_u512/<ipython-input-2-4699ac3a1bfb>.log
Out[2]:
'gnmt_en_vi_u512'

Loading and processing the dataset

The following shows how to process the dataset and cache the processed dataset for future use. The processing steps include the following:

  1. Clipping the source and target sequences
  2. Splitting the string input to a list of tokens
  3. Mapping the string token onto its integer index in the vocabulary
  4. Appending the end-of-sentence (EOS) token to source sentence and adding BOS and EOS tokens to the target sentence

Firstly, we load and cache the dataset with the two helper functions cache_dataset and load_cached_dataset. The functions are straightforward and well commented so no further explanation will be given.

In [3]:
def cache_dataset(dataset, prefix):
    """Cache the processed npy dataset  the dataset into an npz file

    Parameters
    ----------
    dataset : gluon.data.SimpleDataset
    file_path : str
    """
    if not os.path.exists(nmt._constants.CACHE_PATH):
        os.makedirs(nmt._constants.CACHE_PATH)
    src_data = np.concatenate([e[0] for e in dataset])
    tgt_data = np.concatenate([e[1] for e in dataset])
    src_cumlen = np.cumsum([0]+[len(e[0]) for e in dataset])
    tgt_cumlen = np.cumsum([0]+[len(e[1]) for e in dataset])
    np.savez(os.path.join(nmt._constants.CACHE_PATH, prefix + '.npz'),
             src_data=src_data, tgt_data=tgt_data,
             src_cumlen=src_cumlen, tgt_cumlen=tgt_cumlen)


def load_cached_dataset(prefix):
    cached_file_path = os.path.join(nmt._constants.CACHE_PATH, prefix + '.npz')
    if os.path.exists(cached_file_path):
        print('Load cached data from {}'.format(cached_file_path))
        npz_data = np.load(cached_file_path)
        src_data, tgt_data, src_cumlen, tgt_cumlen = [npz_data[n] for n in
                ['src_data', 'tgt_data', 'src_cumlen', 'tgt_cumlen']]
        src_data = np.array([src_data[low:high] for low, high in zip(src_cumlen[:-1], src_cumlen[1:])])
        tgt_data = np.array([tgt_data[low:high] for low, high in zip(tgt_cumlen[:-1], tgt_cumlen[1:])])
        return gluon.data.ArrayDataset(np.array(src_data), np.array(tgt_data))
    else:
        return None

Next, we write the class TrainValDataTransform to have easy access to transforming and clipping the source and target sentences. This class also adds the EOS and BOS tokens for cleaner data. Please refer to the comments in the code for more details.

In [4]:
class TrainValDataTransform(object):
    """Transform the machine translation dataset.

    Clip source and the target sentences to the maximum length. For the source sentence, append the
    EOS. For the target sentence, append BOS and EOS.

    Parameters
    ----------
    src_vocab : Vocab
    tgt_vocab : Vocab
    src_max_len : int
    tgt_max_len : int
    """

    def __init__(self, src_vocab, tgt_vocab, src_max_len, tgt_max_len):
        # On initialization of the class, we set the class variables
        self._src_vocab = src_vocab
        self._tgt_vocab = tgt_vocab
        self._src_max_len = src_max_len
        self._tgt_max_len = tgt_max_len

    def __call__(self, src, tgt):
        # On actual calling of the class, we perform the clipping then the appending of the EOS and BOS tokens.
        if self._src_max_len > 0:
            src_sentence = self._src_vocab[src.split()[:self._src_max_len]]
        else:
            src_sentence = self._src_vocab[src.split()]
        if self._tgt_max_len > 0:
            tgt_sentence = self._tgt_vocab[tgt.split()[:self._tgt_max_len]]
        else:
            tgt_sentence = self._tgt_vocab[tgt.split()]
        src_sentence.append(self._src_vocab[self._src_vocab.eos_token])
        tgt_sentence.insert(0, self._tgt_vocab[self._tgt_vocab.bos_token])
        tgt_sentence.append(self._tgt_vocab[self._tgt_vocab.eos_token])
        src_npy = np.array(src_sentence, dtype=np.int32)
        tgt_npy = np.array(tgt_sentence, dtype=np.int32)
        return src_npy, tgt_npy

We leverage the class written above to create a helper function that processes the dataset in very few lines of code.

In [5]:
def process_dataset(dataset, src_vocab, tgt_vocab, src_max_len=-1, tgt_max_len=-1):
    start = time.time()
    dataset_processed = dataset.transform(TrainValDataTransform(src_vocab, tgt_vocab,
                                                                src_max_len,
                                                                tgt_max_len), lazy=False)
    end = time.time()
    print('Processing time spent: {}'.format(end - start))
    return dataset_processed

Here we define a function load_translation_data that combines all the above steps to load the data, check if it’s been processed, and if not, process the data. The method returns all of the required data for training, validating, and testing our model. Please refer to the comments in the code for more information on what each piece does.

In [6]:
def load_translation_data(dataset, src_lang='en', tgt_lang='vi'):
    """Load translation dataset

    Parameters
    ----------
    dataset : str
    src_lang : str, default 'en'
    tgt_lang : str, default 'vi'

    Returns
    -------
    data_train_processed : Dataset
        The preprocessed training sentence pairs
    data_val_processed : Dataset
        The preprocessed validation sentence pairs
    data_test_processed : Dataset
        The preprocessed test sentence pairs
    val_tgt_sentences : list
        The target sentences in the validation set
    test_tgt_sentences : list
        The target sentences in the test set
    src_vocab : Vocab
        Vocabulary of the source language
    tgt_vocab : Vocab
        Vocabulary of the target language
    """
    common_prefix = 'IWSLT2015_{}_{}_{}_{}'.format(src_lang, tgt_lang,
                                                   src_max_len, tgt_max_len)

    # Load the three datasets from files
    data_train = nlp.data.IWSLT2015('train', src_lang=src_lang, tgt_lang=tgt_lang)
    data_val = nlp.data.IWSLT2015('val', src_lang=src_lang, tgt_lang=tgt_lang)
    data_test = nlp.data.IWSLT2015('test', src_lang=src_lang, tgt_lang=tgt_lang)
    src_vocab, tgt_vocab = data_train.src_vocab, data_train.tgt_vocab
    data_train_processed = load_cached_dataset(common_prefix + '_train')

    # Check if each dataset has been processed or not, and if not, process and cache them.
    if not data_train_processed:
        data_train_processed = process_dataset(data_train, src_vocab, tgt_vocab,
                                               src_max_len, tgt_max_len)
        cache_dataset(data_train_processed, common_prefix + '_train')
    data_val_processed = load_cached_dataset(common_prefix + '_val')
    if not data_val_processed:
        data_val_processed = process_dataset(data_val, src_vocab, tgt_vocab)
        cache_dataset(data_val_processed, common_prefix + '_val')
    data_test_processed = load_cached_dataset(common_prefix + '_test')
    if not data_test_processed:
        data_test_processed = process_dataset(data_test, src_vocab, tgt_vocab)
        cache_dataset(data_test_processed, common_prefix + '_test')

    # Pull out the target sentences for both test and validation
    fetch_tgt_sentence = lambda src, tgt: tgt.split()
    val_tgt_sentences = list(data_val.transform(fetch_tgt_sentence))
    test_tgt_sentences = list(data_test.transform(fetch_tgt_sentence))

    # Return all of the necessary pieces we can extract from the data for training our model
    return data_train_processed, data_val_processed, data_test_processed, \
           val_tgt_sentences, test_tgt_sentences, src_vocab, tgt_vocab

We define a last helper function get_data_lengths to get the length of the datasets, again, for simplified cleaner code later.

In [7]:
def get_data_lengths(dataset):
    return list(dataset.transform(lambda srg, tgt: (len(srg), len(tgt))))

And for the last step of processing, we leverage all of our helper functions to keep the code concise and to these 15-20 lines for use in our main. This does all of the aforementioned processing along with storing the necessary information in memory for training our model.

In [8]:
data_train, data_val, data_test, val_tgt_sentences, test_tgt_sentences, src_vocab, tgt_vocab\
    = load_translation_data(dataset=dataset, src_lang=src_lang, tgt_lang=tgt_lang)
data_train_lengths = get_data_lengths(data_train)
data_val_lengths = get_data_lengths(data_val)
data_test_lengths = get_data_lengths(data_test)

with io.open(os.path.join(save_dir, 'val_gt.txt'), 'w', encoding='utf-8') as of:
    for ele in val_tgt_sentences:
        of.write(' '.join(ele) + '\n')

with io.open(os.path.join(save_dir, 'test_gt.txt'), 'w', encoding='utf-8') as of:
    for ele in test_tgt_sentences:
        of.write(' '.join(ele) + '\n')


data_train = data_train.transform(lambda src, tgt: (src, tgt, len(src), len(tgt)), lazy=False)
data_val = gluon.data.SimpleDataset([(ele[0], ele[1], len(ele[0]), len(ele[1]), i)
                                     for i, ele in enumerate(data_val)])
data_test = gluon.data.SimpleDataset([(ele[0], ele[1], len(ele[0]), len(ele[1]), i)
                                      for i, ele in enumerate(data_test)])
Downloading /root/.mxnet/datasets/iwslt2015/iwslt15.zip from https://apache-mxnet.s3-accelerate.dualstack.amazonaws.com/gluon/dataset/iwslt2015/iwslt15.zip...
Processing time spent: 5.315239191055298
Processing time spent: 0.058951377868652344
Processing time spent: 0.05335402488708496

Sampler and DataLoader construction

Now, we have obtained and stored all of the relevant data information. The next step is to construct the sampler and DataLoader. The first step is to use the batchify function, which pads and stacks sequences to form mini-batches.

In [9]:
train_batchify_fn = nlp.data.batchify.Tuple(nlp.data.batchify.Pad(),
                                            nlp.data.batchify.Pad(),
                                            nlp.data.batchify.Stack(dtype='float32'),
                                            nlp.data.batchify.Stack(dtype='float32'))
test_batchify_fn = nlp.data.batchify.Tuple(nlp.data.batchify.Pad(),
                                           nlp.data.batchify.Pad(),
                                           nlp.data.batchify.Stack(dtype='float32'),
                                           nlp.data.batchify.Stack(dtype='float32'),
                                           nlp.data.batchify.Stack())

We can then construct bucketing samplers, which generate batches by grouping sequences with similar lengths. Here, the bucketing scheme is empirically determined.

In [10]:
bucket_scheme = nlp.data.ExpWidthBucket(bucket_len_step=1.2)
train_batch_sampler = nlp.data.FixedBucketSampler(lengths=data_train_lengths,
                                                  batch_size=batch_size,
                                                  num_buckets=num_buckets,
                                                  shuffle=True,
                                                  bucket_scheme=bucket_scheme)
logging.info('Train Batch Sampler:\n{}'.format(train_batch_sampler.stats()))
val_batch_sampler = nlp.data.FixedBucketSampler(lengths=data_val_lengths,
                                                batch_size=test_batch_size,
                                                num_buckets=num_buckets,
                                                shuffle=False)
logging.info('Valid Batch Sampler:\n{}'.format(val_batch_sampler.stats()))
test_batch_sampler = nlp.data.FixedBucketSampler(lengths=data_test_lengths,
                                                 batch_size=test_batch_size,
                                                 num_buckets=num_buckets,
                                                 shuffle=False)
logging.info('Test Batch Sampler:\n{}'.format(test_batch_sampler.stats()))
2019-07-12 14:51:50,628 - root - Train Batch Sampler:
FixedBucketSampler:
  sample_num=133166, batch_num=1043
  key=[(9, 10), (16, 17), (26, 27), (37, 38), (51, 52)]
  cnt=[11414, 34897, 37760, 23480, 25615]
  batch_size=[128, 128, 128, 128, 128]
2019-07-12 14:51:50,631 - root - Valid Batch Sampler:
FixedBucketSampler:
  sample_num=1553, batch_num=52
  key=[(22, 28), (40, 52), (58, 76), (76, 100), (94, 124)]
  cnt=[1037, 432, 67, 10, 7]
  batch_size=[32, 32, 32, 32, 32]
2019-07-12 14:51:50,633 - root - Test Batch Sampler:
FixedBucketSampler:
  sample_num=1268, batch_num=42
  key=[(23, 29), (43, 53), (63, 77), (83, 101), (103, 125)]
  cnt=[770, 381, 84, 26, 7]
  batch_size=[32, 32, 32, 32, 32]

Given the samplers, we can create a DataLoader, which is iterable. This simply is a data construct (an iterator) that can feed the model batches at a time. For more information refer to this page.

In [11]:
train_data_loader = gluon.data.DataLoader(data_train,
                                          batch_sampler=train_batch_sampler,
                                          batchify_fn=train_batchify_fn,
                                          num_workers=4)
val_data_loader = gluon.data.DataLoader(data_val,
                                        batch_sampler=val_batch_sampler,
                                        batchify_fn=test_batchify_fn,
                                        num_workers=4)
test_data_loader = gluon.data.DataLoader(data_test,
                                         batch_sampler=test_batch_sampler,
                                         batchify_fn=test_batchify_fn,
                                         num_workers=4)

Building the GNMT model

After obtaining the DataLoader, we can finally build the model. The GNMT encoder and decoder can be easily constructed by calling get_gnmt_encoder_decoder function. Then, we feed the encoder and decoder to the NMTModel to construct the GNMT model.

model.hybridize allows computation to be done using the symbolic backend. To understand what it means to be “hybridized,” please refer to this page on MXNet hybridization and its advantages.

In [12]:
encoder, decoder = nmt.gnmt.get_gnmt_encoder_decoder(hidden_size=num_hidden,
                                                     dropout=dropout,
                                                     num_layers=num_layers,
                                                     num_bi_layers=num_bi_layers)
model = nlp.model.translation.NMTModel(src_vocab=src_vocab, tgt_vocab=tgt_vocab, encoder=encoder,
                                       decoder=decoder, embed_size=num_hidden, prefix='gnmt_')
model.initialize(init=mx.init.Uniform(0.1), ctx=ctx)
static_alloc = True
model.hybridize(static_alloc=static_alloc)
logging.info(model)

# Due to the paddings, we need to mask out the losses corresponding to padding tokens.
loss_function = nlp.loss.MaskedSoftmaxCELoss()
loss_function.hybridize(static_alloc=static_alloc)
2019-07-12 14:51:56,367 - root - NMTModel(
  (encoder): GNMTEncoder(
    (dropout_layer): Dropout(p = 0.2, axes=())
    (rnn_cells): HybridSequential(
      (0): BidirectionalCell(forward=LSTMCell(None -> 2048), backward=LSTMCell(None -> 2048))
      (1): LSTMCell(None -> 2048)
    )
  )
  (decoder): GNMTDecoder(
    (attention_cell): DotProductAttentionCell(
      (_dropout_layer): Dropout(p = 0.0, axes=())
      (_proj_query): Dense(None -> 512, linear)
    )
    (dropout_layer): Dropout(p = 0.2, axes=())
    (rnn_cells): HybridSequential(
      (0): LSTMCell(None -> 2048)
      (1): LSTMCell(None -> 2048)
    )
  )
  (src_embed): HybridSequential(
    (0): Embedding(17191 -> 512, float32)
    (1): Dropout(p = 0.0, axes=())
  )
  (tgt_embed): HybridSequential(
    (0): Embedding(7709 -> 512, float32)
    (1): Dropout(p = 0.0, axes=())
  )
  (tgt_proj): Dense(None -> 7709, linear)
)

Here, we build the BeamSearchTranslator and define a predetermined BeamSearchScorer as the heuristical mechanism for the search. For more information on Beam Search and its applications to NLP, check here.

In [13]:
translator = nmt.translation.BeamSearchTranslator(model=model, beam_size=beam_size,
                                                  scorer=nlp.model.BeamSearchScorer(alpha=lp_alpha,
                                                                                    K=lp_k),
                                                  max_length=tgt_max_len + 100)
logging.info('Use beam_size={}, alpha={}, K={}'.format(beam_size, lp_alpha, lp_k))
2019-07-12 14:51:56,383 - root - Use beam_size=10, alpha=1.0, K=5

We define the evaluation function as shown in the code block below. The evaluate function uses the beam search translator to generate outputs for the validation and testing datasets. Please refer to the comments in the code for more information on what each piece does. In addition, we add the write_sentences helper method to easily output the sentences.

In [14]:
def evaluate(data_loader):
    """Evaluate given the data loader

    Parameters
    ----------
    data_loader : gluon.data.DataLoader

    Returns
    -------
    avg_loss : float
        Average loss
    real_translation_out : list of list of str
        The translation output
    """
    translation_out = []
    all_inst_ids = []
    avg_loss_denom = 0
    avg_loss = 0.0

    for _, (src_seq, tgt_seq, src_valid_length, tgt_valid_length, inst_ids) \
            in enumerate(data_loader):
        src_seq = src_seq.as_in_context(ctx)
        tgt_seq = tgt_seq.as_in_context(ctx)
        src_valid_length = src_valid_length.as_in_context(ctx)
        tgt_valid_length = tgt_valid_length.as_in_context(ctx)

        # Calculate Loss
        out, _ = model(src_seq, tgt_seq[:, :-1], src_valid_length, tgt_valid_length - 1)
        loss = loss_function(out, tgt_seq[:, 1:], tgt_valid_length - 1).mean().asscalar()
        all_inst_ids.extend(inst_ids.asnumpy().astype(np.int32).tolist())
        avg_loss += loss * (tgt_seq.shape[1] - 1)
        avg_loss_denom += (tgt_seq.shape[1] - 1)

        # Translate the sequences and score them
        samples, _, sample_valid_length =\
            translator.translate(src_seq=src_seq, src_valid_length=src_valid_length)
        max_score_sample = samples[:, 0, :].asnumpy()
        sample_valid_length = sample_valid_length[:, 0].asnumpy()

        # Iterate through the tokens and stitch the tokens together for the sentence
        for i in range(max_score_sample.shape[0]):
            translation_out.append(
                [tgt_vocab.idx_to_token[ele] for ele in
                 max_score_sample[i][1:(sample_valid_length[i] - 1)]])

    # Calculate the average loss and initialize a None-filled translation list
    avg_loss = avg_loss / avg_loss_denom
    real_translation_out = [None for _ in range(len(all_inst_ids))]

    # Combine all the words/tokens into a sentence for the final translation
    for ind, sentence in zip(all_inst_ids, translation_out):
        real_translation_out[ind] = sentence

    # Return the loss and the translation
    return avg_loss, real_translation_out


def write_sentences(sentences, file_path):
    with io.open(file_path, 'w', encoding='utf-8') as of:
        for sent in sentences:
            of.write(' '.join(sent) + '\n')

Training

Before entering the training stage, we need to create a trainer for updating the parameters based on the loss. In the following example, we create a trainer that uses the ADAM optimizer.

In [15]:
trainer = gluon.Trainer(model.collect_params(), 'adam', {'learning_rate': lr})

We can then write the training loop. During the training, we evaluate on the validation and testing datasets every epoch, and record the parameters that give the highest Bilingual Evaluation Understudy Score (BLEU)[1] score on the validation dataset. Before performing forward and backward computation, we first use the as_in_context function to copy the mini-batch to the GPU. The statement with mx.autograd.record() tells Gluon’s backend to compute the gradients for the part inside the block.

In [16]:
best_valid_bleu = 0.0

# Run through each epoch
for epoch_id in range(epochs):
    log_avg_loss = 0
    log_avg_gnorm = 0
    log_wc = 0
    log_start_time = time.time()

    # Iterate through each batch
    for batch_id, (src_seq, tgt_seq, src_valid_length, tgt_valid_length)\
            in enumerate(train_data_loader):

        src_seq = src_seq.as_in_context(ctx)
        tgt_seq = tgt_seq.as_in_context(ctx)
        src_valid_length = src_valid_length.as_in_context(ctx)
        tgt_valid_length = tgt_valid_length.as_in_context(ctx)

        # Compute gradients and losses
        with mx.autograd.record():
            out, _ = model(src_seq, tgt_seq[:, :-1], src_valid_length, tgt_valid_length - 1)
            loss = loss_function(out, tgt_seq[:, 1:], tgt_valid_length - 1).mean()
            loss = loss * (tgt_seq.shape[1] - 1) / (tgt_valid_length - 1).mean()
            loss.backward()

        grads = [p.grad(ctx) for p in model.collect_params().values()]
        gnorm = gluon.utils.clip_global_norm(grads, clip)
        trainer.step(1)
        src_wc = src_valid_length.sum().asscalar()
        tgt_wc = (tgt_valid_length - 1).sum().asscalar()
        step_loss = loss.asscalar()
        log_avg_loss += step_loss
        log_avg_gnorm += gnorm
        log_wc += src_wc + tgt_wc
        if (batch_id + 1) % log_interval == 0:
            wps = log_wc / (time.time() - log_start_time)
            logging.info('[Epoch {} Batch {}/{}] loss={:.4f}, ppl={:.4f}, gnorm={:.4f}, '
                         'throughput={:.2f}K wps, wc={:.2f}K'
                         .format(epoch_id, batch_id + 1, len(train_data_loader),
                                 log_avg_loss / log_interval,
                                 np.exp(log_avg_loss / log_interval),
                                 log_avg_gnorm / log_interval,
                                 wps / 1000, log_wc / 1000))
            log_start_time = time.time()
            log_avg_loss = 0
            log_avg_gnorm = 0
            log_wc = 0

    # Evaluate the losses on validation and test datasets and find the corresponding BLEU score and log it
    valid_loss, valid_translation_out = evaluate(val_data_loader)
    valid_bleu_score, _, _, _, _ = nmt.bleu.compute_bleu([val_tgt_sentences], valid_translation_out)
    logging.info('[Epoch {}] valid Loss={:.4f}, valid ppl={:.4f}, valid bleu={:.2f}'
                 .format(epoch_id, valid_loss, np.exp(valid_loss), valid_bleu_score * 100))
    test_loss, test_translation_out = evaluate(test_data_loader)
    test_bleu_score, _, _, _, _ = nmt.bleu.compute_bleu([test_tgt_sentences], test_translation_out)
    logging.info('[Epoch {}] test Loss={:.4f}, test ppl={:.4f}, test bleu={:.2f}'
                 .format(epoch_id, test_loss, np.exp(test_loss), test_bleu_score * 100))

    # Output the sentences we predicted on the validation and test datasets
    write_sentences(valid_translation_out,
                    os.path.join(save_dir, 'epoch{:d}_valid_out.txt').format(epoch_id))
    write_sentences(test_translation_out,
                    os.path.join(save_dir, 'epoch{:d}_test_out.txt').format(epoch_id))

    # Save the model if the BLEU score is better than the previous best
    if valid_bleu_score > best_valid_bleu:
        best_valid_bleu = valid_bleu_score
        save_path = os.path.join(save_dir, 'valid_best.params')
        logging.info('Save best parameters to {}'.format(save_path))
        model.save_parameters(save_path)

    # Update the learning rate based on the number of epochs that have passed
    if epoch_id + 1 >= (epochs * 2) // 3:
        new_lr = trainer.learning_rate * lr_update_factor
        logging.info('Learning rate change to {}'.format(new_lr))
        trainer.set_learning_rate(new_lr)
2019-07-12 14:51:59,154 - root - [Epoch 0 Batch 10/1043] loss=7.7375, ppl=2292.6586, gnorm=1.4907, throughput=20.01K wps, wc=54.27K
2019-07-12 14:52:01,126 - root - [Epoch 0 Batch 20/1043] loss=6.3590, ppl=577.6407, gnorm=1.5744, throughput=25.47K wps, wc=50.20K
2019-07-12 14:52:03,624 - root - [Epoch 0 Batch 30/1043] loss=6.3708, ppl=584.5346, gnorm=0.8043, throughput=27.15K wps, wc=67.78K
2019-07-12 14:52:06,035 - root - [Epoch 0 Batch 40/1043] loss=6.1791, ppl=482.5598, gnorm=0.6213, throughput=26.23K wps, wc=63.19K
2019-07-12 14:52:08,412 - root - [Epoch 0 Batch 50/1043] loss=6.1874, ppl=486.5868, gnorm=0.4043, throughput=26.07K wps, wc=61.93K
2019-07-12 14:52:10,660 - root - [Epoch 0 Batch 60/1043] loss=6.1046, ppl=447.9271, gnorm=0.6770, throughput=26.36K wps, wc=59.19K
2019-07-12 14:52:13,339 - root - [Epoch 0 Batch 70/1043] loss=6.1551, ppl=471.1052, gnorm=0.4607, throughput=27.26K wps, wc=72.99K
2019-07-12 14:52:15,795 - root - [Epoch 0 Batch 80/1043] loss=6.0693, ppl=432.3617, gnorm=0.4156, throughput=26.31K wps, wc=64.58K
2019-07-12 14:52:17,896 - root - [Epoch 0 Batch 90/1043] loss=5.9384, ppl=379.3207, gnorm=0.3587, throughput=25.25K wps, wc=53.02K
2019-07-12 14:52:20,119 - root - [Epoch 0 Batch 100/1043] loss=5.8758, ppl=356.3032, gnorm=0.3946, throughput=26.75K wps, wc=59.42K
2019-07-12 14:52:22,551 - root - [Epoch 0 Batch 110/1043] loss=5.8719, ppl=354.9109, gnorm=0.3552, throughput=26.96K wps, wc=65.50K
2019-07-12 14:52:24,818 - root - [Epoch 0 Batch 120/1043] loss=5.8697, ppl=354.1483, gnorm=0.3365, throughput=25.79K wps, wc=58.43K
2019-07-12 14:52:27,273 - root - [Epoch 0 Batch 130/1043] loss=5.9361, ppl=378.4636, gnorm=0.3638, throughput=24.22K wps, wc=59.39K
2019-07-12 14:52:29,671 - root - [Epoch 0 Batch 140/1043] loss=5.8840, ppl=359.2593, gnorm=0.2906, throughput=25.54K wps, wc=61.18K
2019-07-12 14:52:31,969 - root - [Epoch 0 Batch 150/1043] loss=5.8157, ppl=335.5198, gnorm=0.2898, throughput=24.53K wps, wc=56.34K
2019-07-12 14:52:34,272 - root - [Epoch 0 Batch 160/1043] loss=5.7439, ppl=312.2692, gnorm=0.3652, throughput=25.18K wps, wc=57.93K
2019-07-12 14:52:36,715 - root - [Epoch 0 Batch 170/1043] loss=5.7654, ppl=319.0513, gnorm=0.2934, throughput=26.39K wps, wc=64.42K
2019-07-12 14:52:38,566 - root - [Epoch 0 Batch 180/1043] loss=5.4644, ppl=236.1400, gnorm=0.3479, throughput=23.97K wps, wc=44.31K
2019-07-12 14:52:40,988 - root - [Epoch 0 Batch 190/1043] loss=5.6183, ppl=275.4215, gnorm=0.3699, throughput=25.79K wps, wc=62.45K
2019-07-12 14:52:43,128 - root - [Epoch 0 Batch 200/1043] loss=5.5072, ppl=246.4649, gnorm=0.3412, throughput=25.36K wps, wc=54.24K
2019-07-12 14:52:45,193 - root - [Epoch 0 Batch 210/1043] loss=5.3280, ppl=206.0250, gnorm=0.4402, throughput=25.53K wps, wc=52.68K
2019-07-12 14:52:47,183 - root - [Epoch 0 Batch 220/1043] loss=5.3114, ppl=202.6313, gnorm=0.3806, throughput=25.38K wps, wc=50.47K
2019-07-12 14:52:49,507 - root - [Epoch 0 Batch 230/1043] loss=5.3124, ppl=202.8310, gnorm=0.4301, throughput=26.54K wps, wc=61.65K
2019-07-12 14:52:51,685 - root - [Epoch 0 Batch 240/1043] loss=5.3065, ppl=201.6380, gnorm=0.3482, throughput=26.80K wps, wc=58.32K
2019-07-12 14:52:54,280 - root - [Epoch 0 Batch 250/1043] loss=5.4241, ppl=226.8158, gnorm=0.2779, throughput=27.32K wps, wc=70.84K
2019-07-12 14:52:56,552 - root - [Epoch 0 Batch 260/1043] loss=5.2798, ppl=196.3365, gnorm=0.3681, throughput=26.53K wps, wc=60.22K
2019-07-12 14:52:59,221 - root - [Epoch 0 Batch 270/1043] loss=5.3824, ppl=217.5431, gnorm=0.3006, throughput=27.34K wps, wc=72.96K
2019-07-12 14:53:01,564 - root - [Epoch 0 Batch 280/1043] loss=5.2514, ppl=190.8385, gnorm=0.2708, throughput=25.96K wps, wc=60.80K
2019-07-12 14:53:03,416 - root - [Epoch 0 Batch 290/1043] loss=4.9755, ppl=144.8253, gnorm=0.3439, throughput=24.76K wps, wc=45.79K
2019-07-12 14:53:05,666 - root - [Epoch 0 Batch 300/1043] loss=5.0857, ppl=161.6964, gnorm=0.3572, throughput=26.27K wps, wc=59.05K
2019-07-12 14:53:08,061 - root - [Epoch 0 Batch 310/1043] loss=5.1080, ppl=165.3472, gnorm=0.2994, throughput=25.74K wps, wc=61.58K
2019-07-12 14:53:10,152 - root - [Epoch 0 Batch 320/1043] loss=4.9777, ppl=145.1353, gnorm=0.3042, throughput=25.41K wps, wc=53.10K
2019-07-12 14:53:12,467 - root - [Epoch 0 Batch 330/1043] loss=5.0372, ppl=154.0330, gnorm=0.3005, throughput=26.53K wps, wc=61.37K
2019-07-12 14:53:14,733 - root - [Epoch 0 Batch 340/1043] loss=5.0630, ppl=158.0607, gnorm=0.2700, throughput=25.11K wps, wc=56.88K
2019-07-12 14:53:16,889 - root - [Epoch 0 Batch 350/1043] loss=4.9171, ppl=136.6102, gnorm=0.3163, throughput=25.46K wps, wc=54.86K
2019-07-12 14:53:19,265 - root - [Epoch 0 Batch 360/1043] loss=5.0387, ppl=154.2618, gnorm=0.2847, throughput=27.19K wps, wc=64.55K
2019-07-12 14:53:21,814 - root - [Epoch 0 Batch 370/1043] loss=4.9033, ppl=134.7380, gnorm=0.2927, throughput=26.30K wps, wc=66.97K
2019-07-12 14:53:23,922 - root - [Epoch 0 Batch 380/1043] loss=4.8170, ppl=123.5876, gnorm=0.3136, throughput=25.06K wps, wc=52.79K
2019-07-12 14:53:25,965 - root - [Epoch 0 Batch 390/1043] loss=4.7887, ppl=120.1442, gnorm=0.3492, throughput=24.96K wps, wc=50.94K
2019-07-12 14:53:27,869 - root - [Epoch 0 Batch 400/1043] loss=4.6496, ppl=104.5461, gnorm=0.3659, throughput=25.35K wps, wc=48.22K
2019-07-12 14:53:29,884 - root - [Epoch 0 Batch 410/1043] loss=4.8149, ppl=123.3365, gnorm=0.3156, throughput=23.98K wps, wc=48.27K
2019-07-12 14:53:32,116 - root - [Epoch 0 Batch 420/1043] loss=4.8474, ppl=127.4051, gnorm=0.3107, throughput=25.17K wps, wc=56.14K
2019-07-12 14:53:34,689 - root - [Epoch 0 Batch 430/1043] loss=4.8914, ppl=133.1371, gnorm=0.2932, throughput=26.96K wps, wc=69.33K
2019-07-12 14:53:37,184 - root - [Epoch 0 Batch 440/1043] loss=4.7842, ppl=119.6054, gnorm=0.2832, throughput=26.91K wps, wc=67.08K
2019-07-12 14:53:39,272 - root - [Epoch 0 Batch 450/1043] loss=4.6870, ppl=108.5309, gnorm=0.3199, throughput=25.72K wps, wc=53.68K
2019-07-12 14:53:41,219 - root - [Epoch 0 Batch 460/1043] loss=4.4786, ppl=88.1095, gnorm=0.3579, throughput=25.89K wps, wc=50.38K
2019-07-12 14:53:43,553 - root - [Epoch 0 Batch 470/1043] loss=4.7524, ppl=115.8574, gnorm=0.3027, throughput=26.03K wps, wc=60.70K
2019-07-12 14:53:45,639 - root - [Epoch 0 Batch 480/1043] loss=4.3943, ppl=80.9897, gnorm=0.3418, throughput=25.92K wps, wc=54.04K
2019-07-12 14:53:47,511 - root - [Epoch 0 Batch 490/1043] loss=4.5169, ppl=91.5478, gnorm=0.4395, throughput=24.76K wps, wc=46.32K
2019-07-12 14:53:49,481 - root - [Epoch 0 Batch 500/1043] loss=4.5623, ppl=95.7994, gnorm=0.3114, throughput=24.67K wps, wc=48.55K
2019-07-12 14:53:51,223 - root - [Epoch 0 Batch 510/1043] loss=4.3277, ppl=75.7677, gnorm=0.3386, throughput=23.91K wps, wc=41.62K
2019-07-12 14:53:52,897 - root - [Epoch 0 Batch 520/1043] loss=4.2234, ppl=68.2619, gnorm=0.3572, throughput=23.90K wps, wc=39.98K
2019-07-12 14:53:55,126 - root - [Epoch 0 Batch 530/1043] loss=4.6542, ppl=105.0253, gnorm=0.3099, throughput=26.30K wps, wc=58.58K
2019-07-12 14:53:57,133 - root - [Epoch 0 Batch 540/1043] loss=4.5155, ppl=91.4270, gnorm=0.3108, throughput=25.29K wps, wc=50.72K
2019-07-12 14:53:59,449 - root - [Epoch 0 Batch 550/1043] loss=4.5347, ppl=93.1978, gnorm=0.3084, throughput=27.19K wps, wc=62.95K
2019-07-12 14:54:01,321 - root - [Epoch 0 Batch 560/1043] loss=4.3248, ppl=75.5482, gnorm=0.3360, throughput=24.66K wps, wc=46.13K
2019-07-12 14:54:03,613 - root - [Epoch 0 Batch 570/1043] loss=4.3495, ppl=77.4399, gnorm=0.3141, throughput=26.69K wps, wc=61.12K
2019-07-12 14:54:05,751 - root - [Epoch 0 Batch 580/1043] loss=4.3667, ppl=78.7854, gnorm=0.3395, throughput=25.95K wps, wc=55.43K
2019-07-12 14:54:08,375 - root - [Epoch 0 Batch 590/1043] loss=4.5976, ppl=99.2470, gnorm=0.2837, throughput=28.19K wps, wc=73.93K
2019-07-12 14:54:10,581 - root - [Epoch 0 Batch 600/1043] loss=4.4973, ppl=89.7699, gnorm=0.2779, throughput=25.31K wps, wc=55.80K
2019-07-12 14:54:12,660 - root - [Epoch 0 Batch 610/1043] loss=4.3194, ppl=75.1417, gnorm=0.3410, throughput=25.17K wps, wc=52.28K
2019-07-12 14:54:15,296 - root - [Epoch 0 Batch 620/1043] loss=4.5600, ppl=95.5849, gnorm=0.2570, throughput=27.47K wps, wc=72.39K
2019-07-12 14:54:16,827 - root - [Epoch 0 Batch 630/1043] loss=4.0520, ppl=57.5121, gnorm=0.3387, throughput=22.52K wps, wc=34.44K
2019-07-12 14:54:19,066 - root - [Epoch 0 Batch 640/1043] loss=4.3403, ppl=76.7334, gnorm=0.3208, throughput=25.63K wps, wc=57.32K
2019-07-12 14:54:21,530 - root - [Epoch 0 Batch 650/1043] loss=4.4767, ppl=87.9452, gnorm=0.2786, throughput=26.97K wps, wc=66.42K
2019-07-12 14:54:23,394 - root - [Epoch 0 Batch 660/1043] loss=4.1964, ppl=66.4446, gnorm=0.3390, throughput=23.85K wps, wc=44.42K
2019-07-12 14:54:26,203 - root - [Epoch 0 Batch 670/1043] loss=4.5991, ppl=99.3962, gnorm=0.2477, throughput=27.67K wps, wc=77.68K
2019-07-12 14:54:28,489 - root - [Epoch 0 Batch 680/1043] loss=4.4256, ppl=83.5622, gnorm=0.2865, throughput=25.84K wps, wc=59.02K
2019-07-12 14:54:30,480 - root - [Epoch 0 Batch 690/1043] loss=4.2085, ppl=67.2533, gnorm=0.3188, throughput=25.41K wps, wc=50.54K
2019-07-12 14:54:32,553 - root - [Epoch 0 Batch 700/1043] loss=4.2930, ppl=73.1826, gnorm=0.2972, throughput=25.33K wps, wc=52.45K
2019-07-12 14:54:34,327 - root - [Epoch 0 Batch 710/1043] loss=4.1838, ppl=65.6124, gnorm=0.3403, throughput=23.32K wps, wc=41.32K
2019-07-12 14:54:36,345 - root - [Epoch 0 Batch 720/1043] loss=4.2645, ppl=71.1310, gnorm=0.2990, throughput=24.85K wps, wc=50.09K
2019-07-12 14:54:38,388 - root - [Epoch 0 Batch 730/1043] loss=4.1938, ppl=66.2768, gnorm=0.3238, throughput=25.47K wps, wc=52.02K
2019-07-12 14:54:40,694 - root - [Epoch 0 Batch 740/1043] loss=4.3358, ppl=76.3867, gnorm=0.2804, throughput=25.99K wps, wc=59.86K
2019-07-12 14:54:42,810 - root - [Epoch 0 Batch 750/1043] loss=4.1885, ppl=65.9209, gnorm=0.2898, throughput=25.23K wps, wc=53.35K
2019-07-12 14:54:45,345 - root - [Epoch 0 Batch 760/1043] loss=4.2931, ppl=73.1930, gnorm=0.2988, throughput=27.66K wps, wc=70.09K
2019-07-12 14:54:47,162 - root - [Epoch 0 Batch 770/1043] loss=4.1138, ppl=61.1790, gnorm=0.3164, throughput=23.81K wps, wc=43.21K
2019-07-12 14:54:49,906 - root - [Epoch 0 Batch 780/1043] loss=4.3098, ppl=74.4287, gnorm=0.2756, throughput=27.69K wps, wc=75.93K
2019-07-12 14:54:51,840 - root - [Epoch 0 Batch 790/1043] loss=4.1375, ppl=62.6451, gnorm=0.3090, throughput=24.23K wps, wc=46.81K
2019-07-12 14:54:54,124 - root - [Epoch 0 Batch 800/1043] loss=4.2132, ppl=67.5735, gnorm=0.3285, throughput=25.73K wps, wc=58.72K
2019-07-12 14:54:56,328 - root - [Epoch 0 Batch 810/1043] loss=4.0915, ppl=59.8322, gnorm=0.3001, throughput=26.06K wps, wc=57.38K
2019-07-12 14:54:58,505 - root - [Epoch 0 Batch 820/1043] loss=3.9812, ppl=53.5794, gnorm=0.3245, throughput=26.90K wps, wc=58.52K
2019-07-12 14:55:00,690 - root - [Epoch 0 Batch 830/1043] loss=4.1205, ppl=61.5898, gnorm=0.3311, throughput=26.22K wps, wc=57.24K
2019-07-12 14:55:02,743 - root - [Epoch 0 Batch 840/1043] loss=4.0898, ppl=59.7262, gnorm=0.3212, throughput=25.64K wps, wc=52.57K
2019-07-12 14:55:05,081 - root - [Epoch 0 Batch 850/1043] loss=4.1417, ppl=62.9098, gnorm=0.3118, throughput=27.45K wps, wc=64.14K
2019-07-12 14:55:07,193 - root - [Epoch 0 Batch 860/1043] loss=4.1263, ppl=61.9487, gnorm=0.2929, throughput=25.89K wps, wc=54.64K
2019-07-12 14:55:09,654 - root - [Epoch 0 Batch 870/1043] loss=4.1589, ppl=63.9989, gnorm=0.3028, throughput=26.77K wps, wc=65.81K
2019-07-12 14:55:11,715 - root - [Epoch 0 Batch 880/1043] loss=4.0916, ppl=59.8334, gnorm=0.2933, throughput=24.40K wps, wc=50.27K
2019-07-12 14:55:13,929 - root - [Epoch 0 Batch 890/1043] loss=4.1603, ppl=64.0887, gnorm=0.2798, throughput=25.36K wps, wc=56.11K
2019-07-12 14:55:16,238 - root - [Epoch 0 Batch 900/1043] loss=4.1787, ppl=65.2836, gnorm=0.2854, throughput=26.39K wps, wc=60.91K
2019-07-12 14:55:18,304 - root - [Epoch 0 Batch 910/1043] loss=3.9872, ppl=53.9016, gnorm=0.3052, throughput=25.02K wps, wc=51.65K
2019-07-12 14:55:20,614 - root - [Epoch 0 Batch 920/1043] loss=4.1646, ppl=64.3685, gnorm=0.2735, throughput=26.22K wps, wc=60.52K
2019-07-12 14:55:22,479 - root - [Epoch 0 Batch 930/1043] loss=3.9673, ppl=52.8433, gnorm=0.2951, throughput=23.35K wps, wc=43.51K
2019-07-12 14:55:24,424 - root - [Epoch 0 Batch 940/1043] loss=3.9289, ppl=50.8497, gnorm=0.3224, throughput=25.57K wps, wc=49.71K
2019-07-12 14:55:26,976 - root - [Epoch 0 Batch 950/1043] loss=4.2054, ppl=67.0484, gnorm=0.2970, throughput=27.81K wps, wc=70.92K
2019-07-12 14:55:29,629 - root - [Epoch 0 Batch 960/1043] loss=4.1532, ppl=63.6382, gnorm=0.2784, throughput=27.92K wps, wc=74.06K
2019-07-12 14:55:31,478 - root - [Epoch 0 Batch 970/1043] loss=3.8733, ppl=48.1032, gnorm=0.3550, throughput=23.83K wps, wc=44.03K
2019-07-12 14:55:34,153 - root - [Epoch 0 Batch 980/1043] loss=4.1809, ppl=65.4254, gnorm=0.2794, throughput=27.21K wps, wc=72.73K
2019-07-12 14:55:36,573 - root - [Epoch 0 Batch 990/1043] loss=4.0946, ppl=60.0126, gnorm=0.2943, throughput=27.03K wps, wc=65.39K
2019-07-12 14:55:38,761 - root - [Epoch 0 Batch 1000/1043] loss=3.9916, ppl=54.1389, gnorm=0.3094, throughput=25.47K wps, wc=55.65K
2019-07-12 14:55:41,209 - root - [Epoch 0 Batch 1010/1043] loss=4.0750, ppl=58.8517, gnorm=0.2778, throughput=26.82K wps, wc=65.61K
2019-07-12 14:55:43,041 - root - [Epoch 0 Batch 1020/1043] loss=3.8215, ppl=45.6718, gnorm=0.3256, throughput=24.98K wps, wc=45.72K
2019-07-12 14:55:45,225 - root - [Epoch 0 Batch 1030/1043] loss=3.9741, ppl=53.2028, gnorm=0.2978, throughput=25.64K wps, wc=55.97K
2019-07-12 14:55:47,275 - root - [Epoch 0 Batch 1040/1043] loss=3.9365, ppl=51.2392, gnorm=0.3619, throughput=26.02K wps, wc=53.27K
2019-07-12 14:56:04,893 - root - [Epoch 0] valid Loss=2.8535, valid ppl=17.3485, valid bleu=3.13
2019-07-12 14:56:20,435 - root - [Epoch 0] test Loss=2.9888, test ppl=19.8623, test bleu=2.97
2019-07-12 14:56:20,441 - root - Save best parameters to gnmt_en_vi_u512/valid_best.params
2019-07-12 14:56:20,608 - root - Learning rate change to 0.0005

Conclusion

In this notebook, we have shown how to train a GNMT model on the IWSLT 2015 English-Vietnamese dataset using the Gluon NLP toolkit. The complete training script can be found here. The code sequence to reproduce the results can be seen on the machine translation page.

References

[1] Papineni, Kishore, et al. “BLEU: a method for automatic evaluation of machine translation.” Proceedings of the 40th annual meeting on association for computational linguistics. Association for Computational Linguistics, 2002.