Add code for mixture of experts (#521)

modeling and other text generation tasks. It provides reference implementations

of various sequence-to-sequence models, including:

- **Convolutional Neural Networks (CNN)**

  - [Dauphin et al. (2017): Language Modeling with Gated Convolutional Networks](https://arxiv.org/abs/1612.08083)

  - [Gehring et al. (2017): Convolutional Sequence to Sequence Learning](https://arxiv.org/abs/1705.03122)

  - [Edunov et al. (2018): Classical Structured Prediction Losses for Sequence to Sequence Learning](https://arxiv.org/abs/1711.04956)

  - [Fan et al. (2018): Hierarchical Neural Story Generation](https://arxiv.org/abs/1805.04833)

  - [Dauphin et al. (2017): Language Modeling with Gated Convolutional Networks](examples/conv_lm/README.md)

  - [Gehring et al. (2017): Convolutional Sequence to Sequence Learning](examples/conv_seq2seq/README.md)

  - [Edunov et al. (2018): Classical Structured Prediction Losses for Sequence to Sequence Learning](https://github.com/pytorch/fairseq/tree/classic_seqlevel)

  - [Fan et al. (2018): Hierarchical Neural Story Generation](examples/stories/README.md)

- **LightConv and DynamicConv models**

  - **_New_** [Wu et al. (2019): Pay Less Attention with Lightweight and Dynamic Convolutions](https://openreview.net/pdf?id=SkVhlh09tX)

  - **_New_** [Wu et al. (2019): Pay Less Attention with Lightweight and Dynamic Convolutions](examples/pay_less_attention_paper/README.md)

- **Long Short-Term Memory (LSTM) networks**

  - [Luong et al. (2015): Effective Approaches to Attention-based Neural Machine Translation](https://arxiv.org/abs/1508.04025)

  - [Wiseman and Rush (2016): Sequence-to-Sequence Learning as Beam-Search Optimization](https://arxiv.org/abs/1606.02960)

- **Transformer (self-attention) networks**

  - [Vaswani et al. (2017): Attention Is All You Need](https://arxiv.org/abs/1706.03762)

  - [Ott et al. (2018): Scaling Neural Machine Translation](https://arxiv.org/abs/1806.00187)

  - [Ott et al. (2018): Scaling Neural Machine Translation](examples/scaling_nmt/README.md)

  - [Edunov et al. (2018): Understanding Back-Translation at Scale](https://arxiv.org/abs/1808.09381)

  - **_New_** [Shen et al. (2019) Mixture Models for Diverse Machine Translation: Tricks of the Trade](examples/translation_moe/README.md)

Fairseq features:

- multi-GPU (distributed) training on one machine or across multiple machines

- [Language Modeling](examples/language_model/README.md): convolutional models are available

We also have more detailed READMEs to reproduce results from specific papers:

- [Shen et al. (2019) Mixture Models for Diverse Machine Translation: Tricks of the Trade](examples/translation_moe/README.md)

- [Wu et al. (2019): Pay Less Attention with Lightweight and Dynamic Convolutions](examples/pay_less_attention_paper/README.md)

- [Edunov et al. (2018): Classical Structured Prediction Losses for Sequence to Sequence Learning](https://github.com/pytorch/fairseq/tree/classic_seqlevel)

- [Fan et al. (2018): Hierarchical Neural Story Generation](examples/stories/README.md)

$ cd ../..

# Binarize the dataset:

$ TEXT=examples/translation/wmt14_en_de

$ TEXT=examples/translation/wmt17_en_de

$ fairseq-preprocess --source-lang en --target-lang de \

  --trainpref $TEXT/train --validpref $TEXT/valid --testpref $TEXT/test \

  --destdir data-bin/wmt14_en_de --thresholdtgt 0 --thresholdsrc 0

  --destdir data-bin/wmt17_en_de --thresholdtgt 0 --thresholdsrc 0

# Train the model:

# If it runs out of memory, try to set --max-tokens 1500 instead

$ mkdir -p checkpoints/fconv_wmt_en_de

$ fairseq-train data-bin/wmt14_en_de \

$ fairseq-train data-bin/wmt17_en_de \

  --lr 0.5 --clip-norm 0.1 --dropout 0.2 --max-tokens 4000 \

  --criterion label_smoothed_cross_entropy --label-smoothing 0.1 \

  --lr-scheduler fixed --force-anneal 50 \

  --arch fconv_wmt_en_de --save-dir checkpoints/fconv_wmt_en_de

# Generate:

$ fairseq-generate data-bin/wmt14_en_de \

$ fairseq-generate data-bin/wmt17_en_de \

  --path checkpoints/fconv_wmt_en_de/checkpoint_best.pt --beam 5 --remove-bpe

```

    URLS[2]="http://statmt.org/wmt14/training-parallel-nc-v9.tgz"

    FILES[2]="training-parallel-nc-v9.tgz"

    CORPORA[2]="training/news-commentary-v9.de-en"

    OUTDIR=wmt14_en_de

else

    OUTDIR=wmt17_en_de

fi

if [ ! -d "$SCRIPTS" ]; then

src=en

tgt=de

lang=en-de

prep=wmt14_en_de

prep=$OUTDIR

tmp=$prep/tmp

orig=orig

dev=dev/newstest2013

# Mixture Models for Diverse Machine Translation: Tricks of the Trade (Shen et al., 2019)

This page includes instructions for reproducing results from the paper [Mixture Models for Diverse Machine Translation: Tricks of the Trade (Shen et al., 2019)](https://arxiv.org/abs/1902.07816).

## Training a new model on WMT'17 En-De

First, follow the [instructions to download and preprocess the WMT'17 En-De dataset](../translation#prepare-wmt14en2desh).

Make sure to learn a joint vocabulary by passing the `--joined-dictionary` option to `fairseq-preprocess`.

Then we can train a mixture of experts model using the `translation_moe` task.

Use the `--method` option to choose the MoE variant; we support hard mixtures with a learned or uniform prior (`--method hMoElp` and `hMoEup`, respectively) and soft mixures (`--method sMoElp` and `sMoEup`).

To train a hard mixture of experts model with a learned prior (`hMoElp`) on 1 GPU:

```

$ CUDA_VISIBLE_DEVICES=0 fairseq-train data-bin/wmt17_en_de \

  --max-update 100000 \

  --task translation_moe \

  --method hMoElp --mean-pool-gating-network \

  --num-experts 3 \

  --arch transformer_vaswani_wmt_en_de --share-all-embeddings \

  --optimizer adam --adam-betas '(0.9, 0.98)' --clip-norm 0.0 \

  --lr-scheduler inverse_sqrt --warmup-init-lr 1e-07 --warmup-updates 4000 \

  --lr 0.0007 --min-lr 1e-09 \

  --dropout 0.1 --weight-decay 0.0 --criterion cross_entropy \

  --max-tokens 3584 \

  --update-freq 8

```

**Note**: the above command assumes 1 GPU, but accumulates gradients from 8 fwd/bwd passes to simulate training on 8 GPUs.

You can accelerate training on up to 8 GPUs by adjusting the `CUDA_VISIBLE_DEVICES` and `--update-freq` options accordingly.

Once a model is trained, we can generate translations from different experts using the `--gen-expert` option.

For example, to generate from expert 0:

```

$ fairseq-generate data-bin/wmt17_en_de \

    --path checkpoints/checkpoint_best.pt 

    --beam 1 --remove-bpe \

    --task translation_moe \

    --method hMoElp --mean-pool-gating-network \

    --num-experts 3 \

    --gen-expert 0 \

```

You can also use `scripts/score_moe.py` to compute pairwise BLEU and average oracle BLEU.

We'll first download a tokenized version of the multi-reference WMT'14 En-De dataset:

```

$ wget dl.fbaipublicfiles.com/fairseq/data/wmt14-en-de.extra_refs.tok

```

Next apply BPE on the fly and run generation for each expert:

```

$ BPEROOT=examples/translation/subword-nmt/

$ BPE_CODE=examples/translation/wmt17_en_de/code

$ for EXPERT in $(seq 0 2); do \

    cat wmt14-en-de.extra_refs.tok | grep ^S | cut -f 2 | \

      python $BPEROOT/apply_bpe.py -c $BPE_CODE | \

      fairseq-interactive data-bin/wmt17_en_de \

        --path checkpoints/checkpoint_best.pt \

        --beam 1 --remove-bpe \

        --buffer 500 --max-tokens 6000 ; \

        --task translation_moe \

        --method hMoElp --mean-pool-gating-network \

        --num-experts 3 \

        --gen-expert $EXPERT \

  done > wmt14-en-de.extra_refs.tok.gen.3experts

```

Finally compute pairwise BLUE and average oracle BLEU:

```

$ python scripts/score_moe.py --sys wmt14-en-de.extra_refs.tok.gen.3experts --ref wmt14-en-de.extra_refs.tok

pairwise BLEU: 48.26

avg oracle BLEU: 49.50

#refs covered: 2.11

```

This reproduces row 3 from Table 7 in the paper.

## Citation

```bibtex

@article{shen2019mixture,

  title = {Mixture Models for Diverse Machine Translation: Tricks of the Trade},

  author = {Tianxiao Shen and Myle Ott and Michael Auli and Marc'Aurelio Ranzato},

  journal = {arXiv preprint arXiv:1902.07816},

  year = 2019,

}

```

        3) logging outputs to display while training

        """

        net_output = model(**sample['net_input'])

        lprobs = model.get_normalized_probs(net_output, log_probs=True)

        lprobs = lprobs.view(-1, lprobs.size(-1))

        target = model.get_targets(sample, net_output).view(-1)

        loss = F.nll_loss(lprobs, target, size_average=False, ignore_index=self.padding_idx,

                          reduce=reduce)

        loss, _ = self.compute_loss(model, net_output, sample, reduce=reduce)

        sample_size = sample['target'].size(0) if self.args.sentence_avg else sample['ntokens']

        logging_output = {

            'loss': utils.item(loss.data) if reduce else loss.data,

        }

        return loss, sample_size, logging_output

    def compute_loss(self, model, net_output, sample, reduce=True):

        lprobs = model.get_normalized_probs(net_output, log_probs=True)

        lprobs = lprobs.view(-1, lprobs.size(-1))

        target = model.get_targets(sample, net_output).view(-1)

        loss = F.nll_loss(lprobs, target, size_average=False, ignore_index=self.padding_idx,

                          reduce=reduce)

        return loss, loss

    @staticmethod

    def aggregate_logging_outputs(logging_outputs):

        """Aggregate logging outputs from data parallel training."""