Skip to content
Snippets Groups Projects
user avatar
Ning Dong authored
Summary:
Pull Request resolved: https://github.com/pytorch/translate/pull/494

Pull Request resolved: https://github.com/pytorch/fairseq/pull/657

Library side change split from D14924942

Added 2 arguments for load_dataset in PytorchTranslateTask
1. dataset_upsampling. A nested dictionary {direction:{dataset: upsampling_ratio}}. Upsampling_ratio larger than one mean that the bitext is ob- served more often than actually present in the combined bitext and synthetic training corpus.

2. dataset_relative_ratio. A tuple (dataset, ratio). The ratio represents the frequency certain dataset gets sampled to the rest of corpora map.

At most one of them could be specified.

Reviewed By: liezl200

Differential Revision: D15041293

fbshipit-source-id: 92daad29895c234e26d1b19f121106118a3957ad
ff74ca94
History
Name Last commit Last update