Unverified Commit e4c935aa authored by Sergey Edunov's avatar Sergey Edunov Committed by GitHub
Browse files

Merge pull request #91 from facebookresearch/prepare_wmt 

Prepare scripts for WMT14 (#88)
parents fb366144 52b6119a

README.md

+11 −11
Changes for README.md: 11 added lines, 11 removed lines.
Original line number Diff line number Diff line
@@ -49,7 +49,7 @@ The following command-line tools are available: 
## Evaluating Pre-trained Models

First, download a pre-trained model along with its vocabularies:

```

$ curl https://s3.amazonaws.com/fairseq-py/models/wmt14.en-fr.fconv-py.tar.bz2 | tar xvjf -

$ curl https://s3.amazonaws.com/fairseq-py/models/wmt14.v2.en-fr.fconv-py.tar.bz2 | tar xvjf -

```



This model uses a [Byte Pair Encoding (BPE) vocabulary](https://arxiv.org/abs/1508.07909), so we'll have to apply the encoding to the source text before it can be translated.
@@ -135,30 +135,30 @@ BPE continuation markers can be removed with the `--remove-bpe` flag. 


We provide the following pre-trained fully convolutional sequence-to-sequence models:



* [wmt14.en-fr.fconv-py.tar.bz2](https://s3.amazonaws.com/fairseq-py/models/wmt14.en-fr.fconv-py.tar.bz2): Pre-trained model for [WMT14 English-French](http://statmt.org/wmt14/translation-task.html#Download) including vocabularies

* [wmt14.en-de.fconv-py.tar.bz2](https://s3.amazonaws.com/fairseq-py/models/wmt14.en-de.fconv-py.tar.bz2): Pre-trained model for [WMT14 English-German](https://nlp.stanford.edu/projects/nmt) including vocabularies

* [wmt14.en-fr.fconv-py.tar.bz2](https://s3.amazonaws.com/fairseq-py/models/wmt14.v2.en-fr.fconv-py.tar.bz2): Pre-trained model for [WMT14 English-French](http://statmt.org/wmt14/translation-task.html#Download) including vocabularies

* [wmt14.en-de.fconv-py.tar.bz2](https://s3.amazonaws.com/fairseq-py/models/wmt14.v2.en-de.fconv-py.tar.bz2): Pre-trained model for [WMT14 English-German](https://nlp.stanford.edu/projects/nmt) including vocabularies



In addition, we provide pre-processed and binarized test sets for the models above:

* [wmt14.en-fr.newstest2014.tar.bz2](https://s3.amazonaws.com/fairseq-py/data/wmt14.en-fr.newstest2014.tar.bz2): newstest2014 test set for WMT14 English-French

* [wmt14.en-fr.ntst1213.tar.bz2](https://s3.amazonaws.com/fairseq-py/data/wmt14.en-fr.ntst1213.tar.bz2): newstest2012 and newstest2013 test sets for WMT14 English-French

* [wmt14.en-de.newstest2014.tar.bz2](https://s3.amazonaws.com/fairseq-py/data/wmt14.en-de.newstest2014.tar.bz2): newstest2014 test set for WMT14 English-German

* [wmt14.en-fr.newstest2014.tar.bz2](https://s3.amazonaws.com/fairseq-py/data/wmt14.v2.en-fr.newstest2014.tar.bz2): newstest2014 test set for WMT14 English-French

* [wmt14.en-fr.ntst1213.tar.bz2](https://s3.amazonaws.com/fairseq-py/data/wmt14.v2.en-fr.ntst1213.tar.bz2): newstest2012 and newstest2013 test sets for WMT14 English-French

* [wmt14.en-de.newstest2014.tar.bz2](https://s3.amazonaws.com/fairseq-py/data/wmt14.v2.en-de.newstest2014.tar.bz2): newstest2014 test set for WMT14 English-German



Generation with the binarized test sets can be run in batch mode as follows, e.g. for English-French on a GTX-1080ti:

```

$ curl https://s3.amazonaws.com/fairseq-py/models/wmt14.en-fr.fconv-py.tar.bz2 | tar xvjf - -C data-bin

$ curl https://s3.amazonaws.com/fairseq-py/data/wmt14.en-fr.newstest2014.tar.bz2 | tar xvjf - -C data-bin

$ curl https://s3.amazonaws.com/fairseq-py/models/wmt14.v2.en-fr.fconv-py.tar.bz2 | tar xvjf - -C data-bin

$ curl https://s3.amazonaws.com/fairseq-py/data/wmt14.v2.en-fr.newstest2014.tar.bz2 | tar xvjf - -C data-bin

$ python generate.py data-bin/wmt14.en-fr.newstest2014  \

  --path data-bin/wmt14.en-fr.fconv-py/model.pt \

  --beam 5 --batch-size 128 --remove-bpe | tee /tmp/gen.out

...

| Translated 3003 sentences (95451 tokens) in 93.8s (1018.09 tokens/s)

| Generate test with beam=5: BLEU4 = 40.67, 67.7/46.7/34.2/25.3 (BP=1.000, ratio=0.998, syslen=81377, reflen=81194)

| Translated 3003 sentences (96311 tokens) in 166.0s (580.04 tokens/s)

| Generate test with beam=5: BLEU4 = 40.83, 67.5/46.9/34.4/25.5 (BP=1.000, ratio=1.006, syslen=83262, reflen=82787)



# Scoring with score.py:

$ grep ^H /tmp/gen.out | cut -f3- > /tmp/gen.out.sys

$ grep ^T /tmp/gen.out | cut -f2- > /tmp/gen.out.ref

$ python score.py --sys /tmp/gen.out.sys --ref /tmp/gen.out.ref

BLEU4 = 40.67, 67.7/46.7/34.2/25.3 (BP=1.000, ratio=0.998, syslen=81377, reflen=81194)

BLEU4 = 40.83, 67.5/46.9/34.4/25.5 (BP=1.000, ratio=1.006, syslen=83262, reflen=82787)

```



# Join the fairseq community

data/README.md

0 → 100644
+103 −0
Changes for data/README.md: 103 added lines, 0 removed lines.
Original line number Diff line number Diff line 
Sample data processing scripts for the FAIR Sequence-to-Sequence Toolkit



These scripts provide an example of pre-processing data for the NMT task.



# prepare-iwslt14.sh



Provides an example of pre-processing for IWSLT'14 German to English translation task: ["Report on the 11th IWSLT evaluation campaign" by Cettolo et al.](http://workshop2014.iwslt.org/downloads/proceeding.pdf)



Example usage:

```

$ cd data/

$ bash prepare-iwslt14.sh

$ cd ..



# Binarize the dataset:

$ TEXT=data/iwslt14.tokenized.de-en

$ python preprocess.py --source-lang de --target-lang en \

  --trainpref $TEXT/train --validpref $TEXT/valid --testpref $TEXT/test \

  --destdir data-bin/iwslt14.tokenized.de-en



# Train the model:

$ mkdir -p checkpoints/fconv

$ CUDA_VISIBLE_DEVICES=0 python train.py data-bin/iwslt14.tokenized.de-en \

  --lr 0.25 --clip-norm 0.1 --dropout 0.2 --max-tokens 4000 \

  --label-smoothing 0.1 --force-anneal 200 \

  --arch fconv_iwslt_de_en --save-dir checkpoints/fconv



# Generate:

$ python generate.py data-bin/iwslt14.tokenized.de-en \

  --path checkpoints/fconv/checkpoint_best.pt \

  --batch-size 128 --beam 5 --remove-bpe



```





# prepare-wmt14en2de.sh



Provides an example of pre-processing for the WMT'14 English to German translation task. By default it will produce a dataset that was modeled after ["Attention Is All You Need" by Vaswani et al.](https://arxiv.org/abs/1706.03762) that includes news-commentary-v12 data.



To use only data awailable in WMT'14 or to replicate results obtained in the original paper ["Convolutional Sequence to Sequence Learning" by Gehring et al.](https://arxiv.org/abs/1705.03122) run it with --icml17 instead:



```

$ bash prepare-wmt14en2de.sh --icml17

```



Example usage:



```

$ cd data/

$ bash prepare-wmt14en2de.sh

$ cd ..



# Binarize the dataset:

$ TEXT=data/wmt14_en_de

$ python preprocess.py --source-lang en --target-lang de \

  --trainpref $TEXT/train --validpref $TEXT/valid --testpref $TEXT/test \

  --destdir data-bin/wmt14_en_de --thresholdtgt 0 --thresholdsrc 0



# Train the model:

# If it runs out of memory, try to set --max-tokens 1500 instead

$ mkdir -p checkpoints/fconv_wmt_en_de

$ python train.py data-bin/wmt14_en_de \

  --lr 0.5 --clip-norm 0.1 --dropout 0.2 --max-tokens 4000 \

  --label-smoothing 0.1 --force-anneal 50 \

  --arch fconv_wmt_en_de --save-dir checkpoints/fconv_wmt_en_de



# Generate:

$ python generate.py data-bin/wmt14_en_de \

  --path checkpoints/fconv_wmt_en_de/checkpoint_best.pt --beam 5 --remove-bpe



```



# prepare-wmt14en2fr.sh



Provides an example of pre-processing for the WMT'14 English to French translation task.



Example usage:



```

$ cd data/

$ bash prepare-wmt14en2fr.sh

$ cd ..



# Binarize the dataset:

$ TEXT=data/wmt14_en_fr

$ python preprocess.py --source-lang en --target-lang fr \

  --trainpref $TEXT/train --validpref $TEXT/valid --testpref $TEXT/test \

  --destdir data-bin/wmt14_en_fr --thresholdtgt 0 --thresholdsrc 0



# Train the model:

# If it runs out of memory, try to set --max-tokens 1000 instead

$ mkdir -p checkpoints/fconv_wmt_en_fr

$ python train.py data-bin/wmt14_en_fr \

  --lr 0.5 --clip-norm 0.1 --dropout 0.1 --max-tokens 3000 \

  --label-smoothing 0.1 --force-anneal 50 \

  --arch fconv_wmt_en_fr --save-dir checkpoints/fconv_wmt_en_fr



# Generate:

$ python generate.py data-bin/fconv_wmt_en_fr \

  --path checkpoints/fconv_wmt_en_fr/checkpoint_best.pt --beam 5 --remove-bpe



```

data/prepare-wmt14en2de.sh

0 → 100644
+136 −0
Changes for data/prepare-wmt14en2de.sh: 136 added lines, 0 removed lines.
Original line number Diff line number Diff line 
#!/bin/bash

# Adapted from https://github.com/facebookresearch/MIXER/blob/master/prepareData.sh



echo 'Cloning Moses github repository (for tokenization scripts)...'

git clone https://github.com/moses-smt/mosesdecoder.git



echo 'Cloning Subword NMT repository (for BPE pre-processing)...'

git clone https://github.com/rsennrich/subword-nmt.git



SCRIPTS=mosesdecoder/scripts

TOKENIZER=$SCRIPTS/tokenizer/tokenizer.perl

CLEAN=$SCRIPTS/training/clean-corpus-n.perl

NORM_PUNC=$SCRIPTS/tokenizer/normalize-punctuation.perl

REM_NON_PRINT_CHAR=$SCRIPTS/tokenizer/remove-non-printing-char.perl

BPEROOT=subword-nmt

BPE_TOKENS=40000



URLS=(

    "http://statmt.org/wmt13/training-parallel-europarl-v7.tgz"

    "http://statmt.org/wmt13/training-parallel-commoncrawl.tgz"

    "http://data.statmt.org/wmt17/translation-task/training-parallel-nc-v12.tgz"

    "http://statmt.org/wmt14/test-full.tgz"

)

FILES=(

    "training-parallel-europarl-v7.tgz"

    "training-parallel-commoncrawl.tgz"

    "training-parallel-nc-v12.tgz"

    "test-full.tgz"

)

CORPORA=(

    "training/europarl-v7.de-en"

    "commoncrawl.de-en"

    "training/news-commentary-v12.de-en"

)



# This will make the dataset compatible to the one used in "Convolutional Sequence to Sequence Learning"

# https://arxiv.org/abs/1705.03122

if [ "$1" == "--icml17" ]; then

    URLS[2]="http://statmt.org/wmt14/training-parallel-nc-v9.tgz"

    FILES[2]="training-parallel-nc-v9.tgz"

    CORPORA[2]="training/news-commentary-v9.de-en"

fi



if [ ! -d "$SCRIPTS" ]; then

    echo "Please set SCRIPTS variable correctly to point to Moses scripts."

    exit

fi



src=en

tgt=de

lang=en-de

prep=wmt14_en_de

tmp=$prep/tmp

orig=orig



mkdir -p $orig $tmp $prep



cd $orig



for ((i=0;i<${#URLS[@]};++i)); do

    file=${FILES[i]}

    if [ -f $file ]; then

        echo "$file already exists, skipping download"

    else

        url=${URLS[i]}

        wget "$url"

        if [ -f $file ]; then

            echo "$url successfully downloaded."

        else

            echo "$url not successfully downloaded."

            exit -1

        fi

        if [ ${file: -4} == ".tgz" ]; then

            tar zxvf $file

        elif [ ${file: -4} == ".tar" ]; then

            tar xvf $file

        fi

    fi

done

cd ..



echo "pre-processing train data..."

for l in $src $tgt; do

    rm $tmp/train.tags.$lang.tok.$l

    for f in "${CORPORA[@]}"; do

        cat $orig/$f.$l | \

            perl $NORM_PUNC $l | \

            perl $REM_NON_PRINT_CHAR | \

            perl $TOKENIZER -threads 8 -a -l $l >> $tmp/train.tags.$lang.tok.$l

    done

done



echo "pre-processing test data..."

for l in $src $tgt; do

    if [ "$l" == "$src" ]; then

        t="src"

    else

        t="ref"

    fi

    grep '<seg id' $orig/test-full/newstest2014-deen-$t.$l.sgm | \

        sed -e 's/<seg id="[0-9]*">\s*//g' | \

        sed -e 's/\s*<\/seg>\s*//g' | \

        sed -e "s/\’/\'/g" | \

    perl $TOKENIZER -threads 8 -a -l $l > $tmp/test.$l

    echo ""

done



echo "splitting train and valid..."

for l in $src $tgt; do

    awk '{if (NR%100 == 0)  print $0; }' $tmp/train.tags.$lang.tok.$l > $tmp/valid.$l

    awk '{if (NR%100 != 0)  print $0; }' $tmp/train.tags.$lang.tok.$l > $tmp/train.$l

done



TRAIN=$tmp/train.de-en

BPE_CODE=$prep/code

rm -f $TRAIN

for l in $src $tgt; do

    cat $tmp/train.$l >> $TRAIN

done



echo "learn_bpe.py on ${TRAIN}..."

python $BPEROOT/learn_bpe.py -s $BPE_TOKENS < $TRAIN > $BPE_CODE



for L in $src $tgt; do

    for f in train.$L valid.$L test.$L; do

        echo "apply_bpe.py to ${f}..."

        python $BPEROOT/apply_bpe.py -c $BPE_CODE < $tmp/$f > $tmp/bpe.$f

    done

done



perl $CLEAN -ratio 1.5 $tmp/bpe.train $src $tgt $prep/train 1 250

perl $CLEAN -ratio 1.5 $tmp/bpe.valid $src $tgt $prep/valid 1 250



for L in $src $tgt; do

    cp $tmp/bpe.test.$L $prep/test.$L

done

data/prepare-wmt14en2fr.sh

0 → 100644
+136 −0
Changes for data/prepare-wmt14en2fr.sh: 136 added lines, 0 removed lines.
Original line number Diff line number Diff line 
#!/bin/bash

# Adapted from https://github.com/facebookresearch/MIXER/blob/master/prepareData.sh



echo 'Cloning Moses github repository (for tokenization scripts)...'

git clone https://github.com/moses-smt/mosesdecoder.git



echo 'Cloning Subword NMT repository (for BPE pre-processing)...'

git clone https://github.com/rsennrich/subword-nmt.git



SCRIPTS=mosesdecoder/scripts

TOKENIZER=$SCRIPTS/tokenizer/tokenizer.perl

CLEAN=$SCRIPTS/training/clean-corpus-n.perl

NORM_PUNC=$SCRIPTS/tokenizer/normalize-punctuation.perl

REM_NON_PRINT_CHAR=$SCRIPTS/tokenizer/remove-non-printing-char.perl

BPEROOT=subword-nmt

BPE_TOKENS=40000



URLS=(

    "http://statmt.org/wmt13/training-parallel-europarl-v7.tgz"

    "http://statmt.org/wmt13/training-parallel-commoncrawl.tgz"

    "http://statmt.org/wmt13/training-parallel-un.tgz"

    "http://statmt.org/wmt14/training-parallel-nc-v9.tgz"

    "http://statmt.org/wmt10/training-giga-fren.tar"

    "http://statmt.org/wmt14/test-full.tgz"

)

FILES=(

    "training-parallel-europarl-v7.tgz"

    "training-parallel-commoncrawl.tgz"

    "training-parallel-un.tgz"

    "training-parallel-nc-v9.tgz"

    "training-giga-fren.tar"

    "test-full.tgz"

)

CORPORA=(

    "training/europarl-v7.fr-en"

    "commoncrawl.fr-en"

    "un/undoc.2000.fr-en"

    "training/news-commentary-v9.fr-en"

    "giga-fren.release2.fixed"

)



if [ ! -d "$SCRIPTS" ]; then

    echo "Please set SCRIPTS variable correctly to point to Moses scripts."

    exit

fi



src=en

tgt=fr

lang=en-fr

prep=wmt14_en_fr

tmp=$prep/tmp

orig=orig



mkdir -p $orig $tmp $prep



cd $orig



for ((i=0;i<${#URLS[@]};++i)); do

    file=${FILES[i]}

    if [ -f $file ]; then

        echo "$file already exists, skipping download"

    else

        url=${URLS[i]}

        wget "$url"

        if [ -f $file ]; then

            echo "$url successfully downloaded."

        else

            echo "$url not successfully downloaded."

            exit -1

        fi

        if [ ${file: -4} == ".tgz" ]; then

            tar zxvf $file

        elif [ ${file: -4} == ".tar" ]; then

            tar xvf $file

        fi

    fi

done



gunzip giga-fren.release2.fixed.*.gz

cd ..



echo "pre-processing train data..."

for l in $src $tgt; do

    rm $tmp/train.tags.$lang.tok.$l

    for f in "${CORPORA[@]}"; do

        cat $orig/$f.$l | \

            perl $NORM_PUNC $l | \

            perl $REM_NON_PRINT_CHAR | \

            perl $TOKENIZER -threads 8 -a -l $l >> $tmp/train.tags.$lang.tok.$l

    done

done



echo "pre-processing test data..."

for l in $src $tgt; do

    if [ "$l" == "$src" ]; then

        t="src"

    else

        t="ref"

    fi

    grep '<seg id' $orig/test-full/newstest2014-fren-$t.$l.sgm | \

        sed -e 's/<seg id="[0-9]*">\s*//g' | \

        sed -e 's/\s*<\/seg>\s*//g' | \

        sed -e "s/\’/\'/g" | \

    perl $TOKENIZER -threads 8 -a -l $l > $tmp/test.$l

    echo ""

done



echo "splitting train and valid..."

for l in $src $tgt; do

    awk '{if (NR%1333 == 0)  print $0; }' $tmp/train.tags.$lang.tok.$l > $tmp/valid.$l

    awk '{if (NR%1333 != 0)  print $0; }' $tmp/train.tags.$lang.tok.$l > $tmp/train.$l

done



TRAIN=$tmp/train.fr-en

BPE_CODE=$prep/code

rm -f $TRAIN

for l in $src $tgt; do

    cat $tmp/train.$l >> $TRAIN

done



echo "learn_bpe.py on ${TRAIN}..."

python $BPEROOT/learn_bpe.py -s $BPE_TOKENS < $TRAIN > $BPE_CODE



for L in $src $tgt; do

    for f in train.$L valid.$L test.$L; do

        echo "apply_bpe.py to ${f}..."

        python $BPEROOT/apply_bpe.py -c $BPE_CODE < $tmp/$f > $tmp/bpe.$f

    done

done



perl $CLEAN -ratio 1.5 $tmp/bpe.train $src $tgt $prep/train 1 250

perl $CLEAN -ratio 1.5 $tmp/bpe.valid $src $tgt $prep/valid 1 250



for L in $src $tgt; do

    cp $tmp/bpe.test.$L $prep/test.$L

done

fairseq/tokenizer.py

+3 −4
Changes for fairseq/tokenizer.py: 3 added lines, 4 removed lines.
Original line number Diff line number Diff line
@@ -13,12 +13,11 @@ import torch 


from fairseq import dictionary



SPACE_NORMALIZER = re.compile("\s+")



def tokenize_line(line):

    line = re.sub(r"\t", "", line)

    line = re.sub(r"^\s+", "", line)

    line = re.sub(r"\s+$", "", line)

    line = re.sub(r"\s+", " ", line)

    line = SPACE_NORMALIZER.sub(" ", line)

    line = line.strip()

    return line.split()