Skip to content
Commit 42a0150c authored by Louis Martin's avatar Louis Martin Committed by Myle Ott
Browse files

Replace unk with original string 

* Add <eos> for unk replacement
* Add IndexedRawTextDataset to load raw text files
* Replace unk with original string
* Add load_raw_text_dataset() and --output-format
* Move has_binary_files to data.py
parent 7d44181d
Hide whitespace changes
Inline Side-by-side
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment