Unverified Commit 66415206 authored by Myle Ott's avatar Myle Ott Committed by GitHub
Browse files

fairseq-py goes distributed (#106) 

This PR includes breaking API changes to modularize fairseq-py and adds support for distributed training across multiple nodes.

Changes:
- c7033ef: add support for distributed training! See updated README for usage.
- e016299: modularize fairseq-py, adding support for register_model, register_criterion, register_optimizer, etc.
- 154e440: update LSTM implementation to use PackedSequence objects in the encoder, better following best practices and improving perf
- 90c2973 and 1da6265: improve unit test coverage
parent 7e86e30c

README.md

+38 −0
Changes for README.md: 38 added lines, 0 removed lines.
Original line number Diff line number Diff line
@@ -161,6 +161,44 @@ $ python score.py --sys /tmp/gen.out.sys --ref /tmp/gen.out.ref 
BLEU4 = 40.83, 67.5/46.9/34.4/25.5 (BP=1.000, ratio=1.006, syslen=83262, reflen=82787)

```



# Distributed version



Distributed training in fairseq-py is implemented on top of [torch.distributed](http://pytorch.org/docs/master/distributed.html).

In order to run it requires one process per GPU. In order for those processes to be able to discover each other

they need to know a unique host and port that can be used to establish initial connection and each process

needs to be assigned a rank, that is a unique number from 0 to n-1 where n is the total number of GPUs.



Below is the example of training of a big En2Fr model on 16 nodes with 8 GPUs each (in total 128 GPUs):



If you run on a cluster managed by [SLURM](https://slurm.schedmd.com/) you can train the WMT'14 En2Fr model with

the following command:



```

$ DATA=... # path to the preprocessed dataset, must be visible from all nodes

$ PORT=9218 # any available tcp port that can be used by the trained to establish initial connection

$ sbatch --job-name fairseq-py --gres gpu:8 --nodes 16 --ntasks-per-node 8 \

    --cpus-per-task 10 --no-requeue --wrap 'srun --output train.log.node%t \

    --error train.stderr.node%t.%j python train.py $DATA --distributed-world-size 128 \

    --distributed-port $PORT --force-anneal 50 --lr-scheduler fixed --max-epoch 55 \

    --arch fconv_wmt_en_fr --optimizer nag --lr 0.1,4 --max-tokens 3000 \

    --clip-norm 0.1 --dropout 0.1 --criterion label_smoothed_cross_entropy \

    --label-smoothing 0.1 --wd 0.0001'

```



Alternatively you'll need to manually start one process per each GPU:

```

$ DATA=... # path to the preprocessed dataset, must be visible from all nodes

$ HOST_PORT=your.devserver.com:9218 # has to be one of the hosts that will be used by the job \

    and the port on that host has to be available

$ RANK=... # the rank of this process, has to go from 0 to 127 in case of 128 GPUs

$ python train.py $DATA --distributed-world-size 128 \

      --force-anneal 50 --lr-scheduler fixed --max-epoch 55 \

      --arch fconv_wmt_en_fr --optimizer nag --lr 0.1,4 --max-tokens 3000 \

      --clip-norm 0.1 --dropout 0.1 --criterion label_smoothed_cross_entropy \

      --label-smoothing 0.1 --wd 0.0001 \

      --distributed-init-method='tcp://$HOST_PORT' --distributed-rank=$RANK

```



# Join the fairseq community



* Facebook page: https://www.facebook.com/groups/fairseq.users

distributed_train.py

0 → 100644
+45 −0
Changes for distributed_train.py: 45 added lines, 0 removed lines.
Original line number Diff line number Diff line 
#!/usr/bin/env python3 -u

# Copyright (c) 2017-present, Facebook, Inc.

# All rights reserved.

#

# This source code is licensed under the license found in the LICENSE file in

# the root directory of this source tree. An additional grant of patent rights

# can be found in the PATENTS file in the same directory.



import os

import socket

import subprocess



from singleprocess_train import main as single_process_main

from fairseq import distributed_utils, options





def main(args):

    if args.distributed_init_method is None and args.distributed_port > 0:

        # We can determine the init method automatically for Slurm.

        node_list = os.environ.get('SLURM_JOB_NODELIST')

        if node_list is not None:

            try:

                hostnames = subprocess.check_output(['scontrol', 'show', 'hostnames', node_list])

                args.distributed_init_method = 'tcp://{host}:{port}'.format(

                    host=hostnames.split()[0].decode('utf-8'),

                    port=args.distributed_port)

                args.distributed_rank = int(os.environ.get('SLURM_PROCID'))

                args.device_id = int(os.environ.get('SLURM_LOCALID'))

            except subprocess.CalledProcessError as e:  # scontrol failed

                raise e

            except FileNotFoundError as e:  # Slurm is not installed

                pass

    if args.distributed_init_method is None:

        raise ValueError('--distributed-init-method or --distributed-port '

                         'must be specified for distributed training')



    args.distributed_rank = distributed_utils.distributed_init(args)

    print('| initialized host {} as rank {}'.format(socket.gethostname(), args.distributed_rank))

    single_process_main(args)





if __name__ == '__main__':

    parser = options.get_training_parser()

    args = options.parse_args_and_arch(parser)

    main(args)

fairseq/__init__.py

+0 −1
Changes for fairseq/__init__.py: 0 added lines, 1 removed line.
Original line number Diff line number Diff line
@@ -4,7 +4,6 @@ 
# This source code is licensed under the license found in the LICENSE file in

# the root directory of this source tree. An additional grant of patent rights

# can be found in the PATENTS file in the same directory.

#



from .multiprocessing_pdb import pdb

fairseq/bleu.py

+0 −1
Changes for fairseq/bleu.py: 0 added lines, 1 removed line.
Original line number Diff line number Diff line
@@ -4,7 +4,6 @@ 
# This source code is licensed under the license found in the LICENSE file in

# the root directory of this source tree. An additional grant of patent rights

# can be found in the PATENTS file in the same directory.

#



import ctypes

import math

fairseq/clib/temporal_convolution_tbc/temporal_convolution_tbc.cpp

+4 −0
Changes for fairseq/clib/temporal_convolution_tbc/temporal_convolution_tbc.cpp: 4 added lines, 0 removed lines.
Original line number Diff line number Diff line
@@ -20,6 +20,10 @@ at::Type& getDataType(const char* dtype) { 
    return at::getType(at::kCUDA, at::kFloat);

  } else if (strcmp(dtype, "torch.FloatTensor") == 0) {

    return at::getType(at::kCPU, at::kFloat);

  } else if (strcmp(dtype, "torch.cuda.DoubleTensor") == 0) {

    return at::getType(at::kCUDA, at::kDouble);

  } else if (strcmp(dtype, "torch.DoubleTensor") == 0) {

    return at::getType(at::kCPU, at::kDouble);

  } else {

    throw std::runtime_error(std::string("Unsupported data type: ") + dtype);

  }