Code

Preprocess words#198

Code

Open

dgcrouse wants to merge 12 commits intojcjohnson:New_Preprocessjcjohnson/torch-rnn:New_Preprocessfrom dgcrouse:preprocessWordsCopy head branch name to clipboard

+438-92

Commits

Commits on Aug 12, 2016

• Added script that uses words as tokens rather than characters.

  Show description for 69f7f74
  dgcrouse

  committed
  69f7f74

  Copy full SHA for 69f7f74

• Accidentally used a reserved word

  dgcrouse

  committed
  19361a4

  Copy full SHA for 19361a4

Commits on Aug 18, 2016

• Major rewrite to preprocessWords.py, fixed lots of errors, simplified and made more robust.

  dgcrouse

  committed
  53caddd

  Copy full SHA for 53caddd

Commits on Aug 19, 2016

• Fixed multiple issues pertaining to wildcards, e.g. overtraining, also fixed Unicode support

  dgcrouse

  committed
  5e9b1a4

  Copy full SHA for 5e9b1a4

• Added more flexibility for wildcards. Fixed bug where wildcards were capped rather than floored.

  dgcrouse

  committed
  d0fbb4e

  Copy full SHA for d0fbb4e

Commits on Aug 21, 2016

• Added support for seed text in word mode. To ensure constancy of parsing, use the python script scripts/tokenizeWords.py to generate a JSON and then feed that into sample.lua as the start_tokens op…

  Show description for 8190d7f
  dgcrouse

  committed
  8190d7f

  Copy full SHA for 8190d7f

• Updated readme files

  dgcrouse

  committed
  bd5e1a6

  Copy full SHA for bd5e1a6

• Fixed bad link in README.md

  dgcrouse

  committed
  b796c3f

  Copy full SHA for b796c3f

• Fixed additional error with README.md link

  dgcrouse

  committed
  5da9015

  Copy full SHA for 5da9015

Commits on Aug 23, 2016

• Unified word and character preprocessing scripts. Moved original preprocess script to preprocessLegacy.py. Rewrote tokenization script entirely to support new preprocessor output. Updated Readme fi…

  Show description for f77c0d7
  dgcrouse

  committed
  f77c0d7

  Copy full SHA for f77c0d7

• Quick readme fixes

  dgcrouse

  committed
  d69106a

  Copy full SHA for d69106a

Commits on Jun 3, 2017

• Merge branch 'New_Preprocess' into preprocessWords

  dgcrouse

  authored
  45db245

  Copy full SHA for 45db245