Unverified Commit e5b3c1f4 authored by Myle Ott's avatar Myle Ott Committed by GitHub
Browse files

Merge pull request #54: Version 0.1.0 -> 0.2.0 

Release notes:
- 5c7f4954: Added simple LSTM model with input feeding and attention
- 6e4b7e22: Refactored model definitions and incremental generation to be cleaner
- 7ae79c12: Split interactive generation out of generate.py and into a new binary: interactive.py
- 19a3865d: Subtle correctness fix in beam search decoder. Previously, for a beam size of k, we might emit a hypotheses
           if the <eos> was among the top 2*k candidates. Now we only emit hypotheses for which the <eos> is among the
           top-k candidates. This may subtly change generation results, and in the case of k=1 we will now produce
           strictly greedy outputs.
- 97d7fcb9: Fixed bug in padding direction, where previously we right-padded the source and left-padded the target. We
           now left-pad the source and right-pad the target. This should not effect existing trained models, but may
           change (usually improves) the quality of new models.
- f442f896: Add support for batching based on the number of sentences (`--max-sentences`) in addition to the number of
           tokens (`--max-tokens`). When batching by the number of sentences, one can optionally normalize the gradients
           by the number of sentences with `--sentence-avg` (the default is to normalize by the number of tokens).
- c6d6256b: Add `--log-format` option and JSON logger
parents ba5d7dcd 13a3c811

README.md

+11 −12
Changes for README.md: 11 added lines, 12 removed lines.
Original line number Diff line number Diff line
@@ -43,7 +43,7 @@ The following command-line tools are available: 
* `python preprocess.py`: Data pre-processing: build vocabularies and binarize training data

* `python train.py`: Train a new model on one or multiple GPUs

* `python generate.py`: Translate pre-processed data with a trained model

* `python generate.py -i`: Translate raw text with a trained model

* `python interactive.py`: Translate raw text with a trained model

* `python score.py`: BLEU scoring of generated translations against reference translations



## Evaluating Pre-trained Models
@@ -57,22 +57,21 @@ This can be done with the [apply_bpe.py](https://github.com/rsennrich/subword-nm 
`@@` is used as a continuation marker and the original text can be easily recovered with e.g. `sed s/@@ //g` or by passing the `--remove-bpe` flag to `generate.py`.

Prior to BPE, input text needs to be tokenized using `tokenizer.perl` from [mosesdecoder](https://github.com/moses-smt/mosesdecoder).



Let's use `python generate.py -i` to generate translations.

Let's use `python interactive.py` to generate translations interactively.

Here, we use a beam size of 5:

```

$ MODEL_DIR=wmt14.en-fr.fconv-py

$ python generate.py -i \

$ python interactive.py \

 --path $MODEL_DIR/model.pt $MODEL_DIR \

 --beam 5

| loading model(s) from wmt14.en-fr.fconv-py/model.pt

| [en] dictionary: 44206 types

| [fr] dictionary: 44463 types

| model fconv_wmt_en_fr

| loaded checkpoint /private/home/edunov/wmt14.en-fr.fconv-py/model.pt (epoch 37)

| Type the input sentence and press return:

> Why is it rare to discover new marine mam@@ mal species ?

S       Why is it rare to discover new marine mam@@ mal species ?

O       Why is it rare to discover new marine mam@@ mal species ?

H       -0.08662842959165573    Pourquoi est-il rare de découvrir de nouvelles espèces de mammifères marins ?

A       0 1 3 3 5 6 6 10 8 8 8 11 12

H       -0.06429661810398102    Pourquoi est-il rare de découvrir de nouvelles espèces de mammifères marins ?

A       0 1 3 3 5 6 6 8 8 8 7 11 12

```



This generation script produces four types of outputs: a line prefixed with *S* shows the supplied source sentence after applying the vocabulary; *O* is a copy of the original source sentence; *H* is the hypothesis along with an average log-likelihood; and *A* is the attention maxima for each word in the hypothesis, including the end-of-sentence marker which is omitted from the text.
@@ -114,7 +113,7 @@ Also note that the batch size is specified in terms of the maximum number of tok 
You may need to use a smaller value depending on the available GPU memory on your system.



### Generation

Once your model is trained, you can generate translations using `python generate.py` **(for binarized data)** or `python generate.py -i` **(for raw text)**:

Once your model is trained, you can generate translations using `python generate.py` **(for binarized data)** or `python interactive.py` **(for raw text)**:

```

$ python generate.py data-bin/iwslt14.tokenized.de-en \

  --path checkpoints/fconv/checkpoint_best.pt \
@@ -152,14 +151,14 @@ $ python generate.py data-bin/wmt14.en-fr.newstest2014 \ 
  --path data-bin/wmt14.en-fr.fconv-py/model.pt \

  --beam 5 --batch-size 128 --remove-bpe | tee /tmp/gen.out

...

| Translated 3003 sentences (95451 tokens) in 94.5s (1009.60 tokens/s)

| Generate test with beam=5: BLEU4 = 40.70, 67.7/46.8/34.2/25.4 (BP=1.000, ratio=1.000, syslen=81190, reflen=81194)

| Translated 3003 sentences (95451 tokens) in 93.8s (1018.09 tokens/s)

| Generate test with beam=5: BLEU4 = 40.67, 67.7/46.7/34.2/25.3 (BP=1.000, ratio=0.998, syslen=81377, reflen=81194)



# Scoring with score.py:

$ grep ^H /tmp/gen.out | cut -f3- > /tmp/gen.out.sys

$ grep ^T /tmp/gen.out | cut -f2- > /tmp/gen.out.ref

$ python score.py --sys /tmp/gen.out.sys --ref /tmp/gen.out.ref

BLEU4 = 40.70, 67.7/46.8/34.2/25.4 (BP=1.000, ratio=1.000, syslen=81190, reflen=81194)

BLEU4 = 40.67, 67.7/46.7/34.2/25.3 (BP=1.000, ratio=0.998, syslen=81377, reflen=81194)

```



# Join the fairseq community

fairseq/criterions/__init__.py

+0 −1
Changes for fairseq/criterions/__init__.py: 0 added lines, 1 removed line.
Original line number Diff line number Diff line
@@ -7,7 +7,6 @@ 
#



from .cross_entropy import CrossEntropyCriterion

from .fairseq_criterion import FairseqCriterion

from .label_smoothed_cross_entropy import LabelSmoothedCrossEntropyCriterion



__all__ = [

fairseq/criterions/cross_entropy.py

+3 −4
Changes for fairseq/criterions/cross_entropy.py: 3 added lines, 4 removed lines.
Original line number Diff line number Diff line
@@ -14,9 +14,8 @@ from .fairseq_criterion import FairseqCriterion 


class CrossEntropyCriterion(FairseqCriterion):



    def __init__(self, padding_idx):

        super().__init__()

        self.padding_idx = padding_idx

    def __init__(self, args, dst_dict):

        super().__init__(args, dst_dict)



    def forward(self, model, sample):

        """Compute the loss for the given sample.
@@ -30,7 +29,7 @@ class CrossEntropyCriterion(FairseqCriterion): 
        input = net_output.view(-1, net_output.size(-1))

        target = sample['target'].view(-1)

        loss = F.cross_entropy(input, target, size_average=False, ignore_index=self.padding_idx)

        sample_size = sample['ntokens']

        sample_size = sample['target'].size(0) if self.args.sentence_avg else sample['ntokens']

        logging_output = {

            'loss': loss.data[0],

            'sample_size': sample_size,

fairseq/criterions/fairseq_criterion.py

+3 −1
Changes for fairseq/criterions/fairseq_criterion.py: 3 added lines, 1 removed line.
Original line number Diff line number Diff line
@@ -11,8 +11,10 @@ from torch.nn.modules.loss import _Loss 


class FairseqCriterion(_Loss):



    def __init__(self):

    def __init__(self, args, dst_dict):

        super().__init__()

        self.args = args

        self.padding_idx = dst_dict.pad()



    def forward(self, model, sample):

        """Compute the loss for the given sample.

fairseq/criterions/label_smoothed_cross_entropy.py

+6 −7
Changes for fairseq/criterions/label_smoothed_cross_entropy.py: 6 added lines, 7 removed lines.
Original line number Diff line number Diff line
@@ -14,7 +14,7 @@ import torch.nn.functional as F 
from .fairseq_criterion import FairseqCriterion





class LabelSmoothedCrossEntropy(torch.autograd.Function):

class LabelSmoothedNLLLoss(torch.autograd.Function):



    @staticmethod

    def forward(ctx, input, target, eps, padding_idx, weights):
@@ -43,10 +43,9 @@ class LabelSmoothedCrossEntropy(torch.autograd.Function): 


class LabelSmoothedCrossEntropyCriterion(FairseqCriterion):



    def __init__(self, eps, padding_idx=None, weights=None):

        super().__init__()

        self.eps = eps

        self.padding_idx = padding_idx

    def __init__(self, args, dst_dict, weights=None):

        super().__init__(args, dst_dict)

        self.eps = args.label_smoothing

        self.weights = weights



    def forward(self, model, sample):
@@ -60,8 +59,8 @@ class LabelSmoothedCrossEntropyCriterion(FairseqCriterion): 
        net_output = model(**sample['net_input'])

        input = F.log_softmax(net_output.view(-1, net_output.size(-1)))

        target = sample['target'].view(-1)

        loss = LabelSmoothedCrossEntropy.apply(input, target, self.eps, self.padding_idx, self.weights)

        sample_size = sample['ntokens']

        loss = LabelSmoothedNLLLoss.apply(input, target, self.eps, self.padding_idx, self.weights)

        sample_size = sample['target'].size(0) if self.args.sentence_avg else sample['ntokens']

        logging_output = {

            'loss': loss.data[0],

            'sample_size': sample_size,