Commit 88bf8b56 authored by James King's avatar James King Committed by Facebook Github Bot
Browse files

Fixed the issue that no space in string converted from tensor 

Summary: Pull Request resolved: https://github.com/pytorch/fairseq/pull/548

Differential Revision: D14286021

Pulled By: myleott

fbshipit-source-id: 7c725304185e63787220371a812ec860e178872c
parent 66262a38

fairseq/data/data_utils.py

+1 −1
Original line number Diff line number Diff line
@@ -176,7 +176,7 @@ def batch_by_size( 


def process_bpe_symbol(sentence: str, bpe_symbol: str):

    if bpe_symbol == 'sentencepiece':

        sentence = sentence.replace('\u2581', ' ').strip()

        sentence = sentence.replace(' ','').replace('\u2581', ' ').strip()

    elif bpe_symbol is not None:

        sentence = (sentence + ' ').replace(bpe_symbol, '').rstrip()

    return sentence

fairseq/data/dictionary.py

+1 −1

File changed.

Contains only whitespace changes.