README-list.md

DPK Python Transforms

installation

The transforms are delivered as a standard pyton library available on pypi and can be installed using pip install:

python -m pip install data-prep-toolkit-transforms[all] or python -m pip install data-prep-toolkit-transforms[ray, all] or python -m pip install data-prep-toolkit-transforms[language]

installing the python transforms will also install data-prep-toolkit

installing the ray transforms will also install data-prep-toolkit[ray]

List of Transforms in current package

Note: This list includes the transforms that were part of the release starting with data-prep-toolkit-transforms:0.2.1. This list may not always reflect up to date information. Users are encourage to raise an issue in git when they discover missing components or packages that are listed below but not in the current release they get from pypi.

• code
  • code2parquet
  • header_cleanser (Not available on MacOS)
  • code_quality
  • proglang_select
  • code_profiler
• language
  • doc_chunk
  • doc_quality
  • lang_id
  • pdf2parquet
  • text_encoder
  • pii_redactor
• universal
  • ededup
  • fdedup
  • filter
  • resize
  • tokenization
  • doc_id
  • web2parquet

Release notes:

1.0.1.dev1

Added Gneissweb transforms
fdedup fix for windows

1.0.1.dev0

PR #979 (code_profiler)

1.0.0.a6

Added Profiler
Added Resize

1.0.0.a5

Added Pii Redactor
Relax fasttext requirement >= 0.9.2

1.0.0.a4

Added missing ray implementation for lang_id, doc_quality, tokenization and filter
Added ray notebooks for lang id, Doc Quality, tokenization, and Filter

1.0.0.a3

Added code_profiler

1.0.0.a2

Relax dependencies on pandas (use latest or whatever is installed by application) Relax dependencies on requests (use latest or whatever is installed by application)