Commit 7e60d45b authored by Liezl Puzon's avatar Liezl Puzon Committed by Facebook Github Bot
Browse files

Support for BPE vocabs + denoising autoencoder in PyTorch Translate (#362) 

Summary:
Pull Request resolved: https://github.com/pytorch/fairseq/pull/362

Pull Request resolved: https://github.com/pytorch/translate/pull/254

This actually uses the fairseq logic which supports BPE cont / end word marker suffixes.

Reviewed By: xianxl

Differential Revision: D12952766

fbshipit-source-id: 35a1bbc38240e4145bec0fc419f2d0a6a73ae2e5
parent 880e7cd4

fairseq/data/noising.py

+1 −1
Original line number Diff line number Diff line
@@ -83,7 +83,7 @@ class WordDropout(WordNoising): 
        assert 0 < dropout_prob < 1



        # be sure to drop entire words

        word_idx = self._get_bpe_word_idx(x)

        word_idx = self.get_word_idx(x)

        sentences = []

        modified_lengths = []

        for i in range(lengths.size(0)):