Add fairseq to PyPI (#495)

If you use Docker make sure to increase the shared memory size either with

`--ipc=host` or `--shm-size` as command line options to `nvidia-docker run`.

After PyTorch is installed, you can install fairseq with:

After PyTorch is installed, you can install fairseq with `pip`:

```

pip install -r requirements.txt

python setup.py build develop

pip install fairseq

```

**Installing from source**

To install fairseq from source and develop locally:

```

git clone https://github.com/pytorch/fairseq

cd fairseq

pip install --editable .

```

# Getting Started

Fairseq provides several command-line tools for training and evaluating models:

- :ref:`preprocess.py`: Data pre-processing: build vocabularies and binarize training data

- :ref:`train.py`: Train a new model on one or multiple GPUs

- :ref:`generate.py`: Translate pre-processed data with a trained model

- :ref:`interactive.py`: Translate raw text with a trained model

- :ref:`score.py`: BLEU scoring of generated translations against reference translations

- :ref:`eval_lm.py`: Language model evaluation

- :ref:`fairseq-preprocess`: Data pre-processing: build vocabularies and binarize training data

- :ref:`fairseq-train`: Train a new model on one or multiple GPUs

- :ref:`fairseq-generate`: Translate pre-processed data with a trained model

- :ref:`fairseq-interactive`: Translate raw text with a trained model

- :ref:`fairseq-score`: BLEU scoring of generated translations against reference translations

- :ref:`fairseq-eval-lm`: Language model evaluation

.. _preprocess.py:

.. _fairseq-preprocess:

preprocess.py

~~~~~~~~~~~~~

fairseq-preprocess

~~~~~~~~~~~~~~~~~~

.. automodule:: preprocess

    .. argparse::

        :module: preprocess

        :func: get_parser

        :prog: preprocess.py

        :module: fairseq.options

        :func: get_preprocessing_parser

        :prog: fairseq-preprocess

.. _train.py:

.. _fairseq-train:

train.py

~~~~~~~~

fairseq-train

~~~~~~~~~~~~~

.. automodule:: train

    .. argparse::

        :module: fairseq.options

        :func: get_training_parser

        :prog: train.py

        :prog: fairseq-train

.. _generate.py:

.. _fairseq-generate:

generate.py

~~~~~~~~~~~

fairseq-generate

~~~~~~~~~~~~~~~~

.. automodule:: generate

    .. argparse::

        :module: fairseq.options

        :func: get_generation_parser

        :prog: generate.py

        :prog: fairseq-generate

.. _interactive.py:

.. _fairseq-interactive:

interactive.py

~~~~~~~~~~~~~~

fairseq-interactive

~~~~~~~~~~~~~~~~~~~

.. automodule:: interactive

    .. argparse::

        :module: fairseq.options

        :func: get_interactive_generation_parser

        :prog: interactive.py

        :prog: fairseq-interactive

.. _score.py:

.. _fairseq-score:

score.py

~~~~~~~~

fairseq-score

~~~~~~~~~~~~~

.. automodule:: score

    .. argparse::

        :module: score

        :module: fairseq_cli.score

        :func: get_parser

        :prog: score.py

        :prog: fairseq-score

.. _eval_lm.py:

.. _fairseq-eval-lm:

eval_lm.py

~~~~~~~~~~

fairseq-eval-lm

~~~~~~~~~~~~~~~

.. automodule:: eval_lm

    .. argparse::

        :module: fairseq.options

        :func: get_eval_lm_parser

        :prog: eval_lm.py

        :prog: fairseq-eval-lm

# built documents.

#

# The short X.Y version.

version = '0.6.0'

version = '0.6.1'

# The full version, including alpha/beta/rc tags.

release = '0.6.0'

release = '0.6.1'

# The language for content autogenerated by Sphinx. Refer to documentation

# for a list of supported languages.

Iterators

---------

.. autoclass:: fairseq.data.BufferedIterator

    :members:

.. autoclass:: fairseq.data.CountingIterator

    :members:

.. autoclass:: fairseq.data.EpochBatchIterator

script using the ``wmt14.en-fr.fconv-cuda/bpecodes`` file. ``@@`` is

used as a continuation marker and the original text can be easily

recovered with e.g. ``sed s/@@ //g`` or by passing the ``--remove-bpe``

flag to :ref:`generate.py`. Prior to BPE, input text needs to be tokenized

flag to :ref:`fairseq-generate`. Prior to BPE, input text needs to be tokenized

using ``tokenizer.perl`` from

`mosesdecoder <https://github.com/moses-smt/mosesdecoder>`__.

Let's use :ref:`interactive.py` to generate translations

Let's use :ref:`fairseq-interactive` to generate translations

interactively. Here, we use a beam size of 5:

.. code-block:: console

    > MODEL_DIR=wmt14.en-fr.fconv-py

    > python interactive.py \

    > fairseq-interactive \

        --path $MODEL_DIR/model.pt $MODEL_DIR \

        --beam 5 --source-lang en --target-lang fr

    | loading model(s) from wmt14.en-fr.fconv-py/model.pt

    > bash prepare-iwslt14.sh

    > cd ../..

    > TEXT=examples/translation/iwslt14.tokenized.de-en

    > python preprocess.py --source-lang de --target-lang en \

    > fairseq-preprocess --source-lang de --target-lang en \

        --trainpref $TEXT/train --validpref $TEXT/valid --testpref $TEXT/test \

        --destdir data-bin/iwslt14.tokenized.de-en

Training

--------

Use :ref:`train.py` to train a new model. Here a few example settings that work

Use :ref:`fairseq-train` to train a new model. Here a few example settings that work

well for the IWSLT 2014 dataset:

.. code-block:: console

    > mkdir -p checkpoints/fconv

    > CUDA_VISIBLE_DEVICES=0 python train.py data-bin/iwslt14.tokenized.de-en \

    > CUDA_VISIBLE_DEVICES=0 fairseq-train data-bin/iwslt14.tokenized.de-en \

        --lr 0.25 --clip-norm 0.1 --dropout 0.2 --max-tokens 4000 \

        --arch fconv_iwslt_de_en --save-dir checkpoints/fconv

By default, :ref:`train.py` will use all available GPUs on your machine. Use the

By default, :ref:`fairseq-train` will use all available GPUs on your machine. Use the

``CUDA_VISIBLE_DEVICES`` environment variable to select specific GPUs and/or to

change the number of GPU devices that will be used.

----------

Once your model is trained, you can generate translations using

:ref:`generate.py` **(for binarized data)** or

:ref:`interactive.py` **(for raw text)**:

:ref:`fairseq-generate` **(for binarized data)** or

:ref:`fairseq-interactive` **(for raw text)**:

.. code-block:: console

    > python generate.py data-bin/iwslt14.tokenized.de-en \

    > fairseq-generate data-bin/iwslt14.tokenized.de-en \

        --path checkpoints/fconv/checkpoint_best.pt \

        --batch-size 128 --beam 5

    | [de] dictionary: 35475 types

.. code-block:: console

    > CUDA_VISIBLE_DEVICES=0 python train.py --update-freq 8 (...)

    > CUDA_VISIBLE_DEVICES=0 fairseq-train --update-freq 8 (...)

Training with half precision floating point (FP16)

--------------------------------------------------

.. code-block:: console

    > python train.py --fp16 (...)

    > fairseq-train --fp16 (...)

Lazily loading large training datasets

--------------------------------------

    > python -m torch.distributed.launch --nproc_per_node=8 \

        --nnodes=2 --node_rank=0 --master_addr="192.168.1.1" \

        --master_port=1234 \

        train.py data-bin/wmt16_en_de_bpe32k \

        $(which fairseq-train) data-bin/wmt16_en_de_bpe32k \

        --arch transformer_vaswani_wmt_en_de_big --share-all-embeddings \

        --optimizer adam --adam-betas '(0.9, 0.98)' --clip-norm 0.0 \

        --lr-scheduler inverse_sqrt --warmup-init-lr 1e-07 --warmup-updates 4000 \