Merge internal changes (#283)

Fairseq features:

- multi-GPU (distributed) training on one machine or across multiple machines

- fast beam search generation on both CPU and GPU

- fast generation on both CPU and GPU with multiple search algorithms implemented:

  - beam search

  - Diverse Beam Search ([Vijayakumar et al., 2016](https://arxiv.org/abs/1610.02424))

  - sampling (unconstrained and top-k)

- large mini-batch training even on a single GPU via delayed updates

- fast half-precision floating point (FP16) training

- extensible: easily register new models, criterions, and tasks

- extensible: easily register new models, criterions, tasks, optimizers and learning rate schedulers

We also provide [pre-trained models](#pre-trained-models) for several benchmark

translation and language modeling datasets.

* For training new models, you'll also need an NVIDIA GPU and [NCCL](https://github.com/NVIDIA/nccl)

* Python version 3.6

Currently fairseq requires PyTorch version >= 0.4.0.

Currently fairseq requires PyTorch version >= 1.0.0.

Please follow the instructions here: https://github.com/pytorch/pytorch#installation.

If you use Docker make sure to increase the shared memory size either with

#!/usr/bin/env python3 -u

# Copyright (c) 2017-present, Facebook, Inc.

# All rights reserved.

#

# This source code is licensed under the license found in the LICENSE file in

# the root directory of this source tree. An additional grant of patent rights

# can be found in the PATENTS file in the same directory.

import os

import socket

import subprocess

from train import main as single_process_main

from fairseq import distributed_utils, options

def main(args):

    if args.distributed_init_method is None and args.distributed_port > 0:

        # We can determine the init method automatically for Slurm.

        node_list = os.environ.get('SLURM_JOB_NODELIST')

        if node_list is not None:

            try:

                hostnames = subprocess.check_output(['scontrol', 'show', 'hostnames', node_list])

                args.distributed_init_method = 'tcp://{host}:{port}'.format(

                    host=hostnames.split()[0].decode('utf-8'),

                    port=args.distributed_port)

                args.distributed_rank = int(os.environ.get('SLURM_PROCID'))

                args.device_id = int(os.environ.get('SLURM_LOCALID'))

            except subprocess.CalledProcessError as e:  # scontrol failed

                raise e

            except FileNotFoundError as e:  # Slurm is not installed

                pass

    if args.distributed_init_method is None and args.distributed_port is None:

        raise ValueError('--distributed-init-method or --distributed-port '

                         'must be specified for distributed training')

    args.distributed_rank = distributed_utils.distributed_init(args)

    print('| initialized host {} as rank {}'.format(socket.gethostname(), args.distributed_rank))

    single_process_main(args)

if __name__ == '__main__':

    parser = options.get_training_parser()

    args = options.parse_args_and_arch(parser)

    main(args)

Criterions

==========

Criterions compute the loss function given the model and batch, roughly::

  loss = criterion(model, batch)

.. automodule:: fairseq.criterions

    :members:

.. autoclass:: fairseq.criterions.FairseqCriterion

    :members:

    :undoc-members:

.. autoclass:: fairseq.criterions.adaptive_loss.AdaptiveLoss

    :members:

    :undoc-members:

.. autoclass:: fairseq.criterions.composite_loss.CompositeLoss

    :members:

    :undoc-members:

.. autoclass:: fairseq.criterions.cross_entropy.CrossEntropyCriterion

    :members:

    :undoc-members:

.. autoclass:: fairseq.criterions.label_smoothed_cross_entropy.LabelSmoothedCrossEntropyCriterion

    :members:

    :undoc-members:

.. autoclass:: fairseq.data.MonolingualDataset

    :members:

**Helper Datasets**

These datasets wrap other :class:`fairseq.data.FairseqDataset` instances and

provide additional functionality:

.. autoclass:: fairseq.data.BacktranslationDataset

    :members:

.. autoclass:: fairseq.data.ConcatDataset

    :members:

.. autoclass:: fairseq.data.RoundRobinZipDatasets

    :members:

.. autoclass:: fairseq.data.TransformEosDataset

    :members:

Dictionary

----------

Iterators

---------

.. autoclass:: fairseq.data.BufferedIterator

    :members:

.. autoclass:: fairseq.data.CountingIterator

    :members:

.. autoclass:: fairseq.data.EpochBatchIterator

    > MODEL_DIR=wmt14.en-fr.fconv-py

    > python interactive.py \

        --path $MODEL_DIR/model.pt $MODEL_DIR \

        --beam 5

        --beam 5 --source-lang en --target-lang fr

    | loading model(s) from wmt14.en-fr.fconv-py/model.pt

    | [en] dictionary: 44206 types

    | [fr] dictionary: 44463 types

    | Type the input sentence and press return:

    > Why is it rare to discover new marine mam@@ mal species ?

    O       Why is it rare to discover new marine mam@@ mal species ?

    H       -0.06429661810398102    Pourquoi est-il rare de découvrir de nouvelles espèces de mammifères marins ?

    A       0 1 3 3 5 6 6 8 8 8 7 11 12

This generation script produces four types of outputs: a line prefixed

with *S* shows the supplied source sentence after applying the

vocabulary; *O* is a copy of the original source sentence; *H* is the

hypothesis along with an average log-likelihood; and *A* is the

attention maxima for each word in the hypothesis, including the

    H       -0.1525060087442398     Pourquoi est @-@ il rare de découvrir de nouvelles espèces de mammifères marins ?

    P       -0.2221 -0.3122 -0.1289 -0.2673 -0.1711 -0.1930 -0.1101 -0.1660 -0.1003 -0.0740 -0.1101 -0.0814 -0.1238 -0.0985 -0.1288

This generation script produces three types of outputs: a line prefixed

with *O* is a copy of the original source sentence; *H* is the

hypothesis along with an average log-likelihood; and *P* is the

positional score per token position, including the

end-of-sentence marker which is omitted from the text.

See the `README <https://github.com/pytorch/fairseq#pre-trained-models>`__ for a