Skip to content

Releases: JohnSnowLabs/spark-nlp

Spark NLP 4.3.0: New HuBERT for speech recognition, new Swin Transformer for Image Classification, new Zero-shot annotator for Entity Recognition, CamemBERT for question answering, new Databricks and EMR with support for Spark 3.3, 1000+ state-of-the-art models and many more!

09 Feb 19:45
@maziyarpanahi maziyarpanahi
4.3.0
6cba0fe
Compare
Choose a tag to compare
Spark NLP 4.3.0: New HuBERT for speech recognition, new Swin Transformer for Image Classification, new Zero-shot annotator for Entity Recognition, CamemBERT for question answering, new Databricks and EMR with support for Spark 3.3, 1000+ state-of-the-art models and many more!

πŸ“’ Overview

We are very excited to release Spark NLP πŸš€ 4.3.0! This has been one of the biggest releases we have ever done and we are so proud to share this with our community! πŸŽ‰

This release extends support for another Image Classification by introducing Swin Transformer, also extending support for speech recognition by introducing HuBERT annotator, a brand new modern extractive transformer-based Question answering (QA) annotator for tasks like SQuAD based on CamemBERT architecture, new Databricks & EMR with support for Spark 3.3, 1000+ state-of-the-art models, and many more enhancements and bug fixes!

We are also celebrating crossing 12600+ free and open-source models & pipelines in our Models Hub. πŸŽ‰ As always, we would like to thank our community for their feedback, questions, and feature requests.

πŸ”₯ New Features

HuBERT

NEW: Introducing HubertForCTC annotator in Spark NLP πŸš€. HubertForCTC can load HuBERT models that match or surpasses the SOTA approaches for speech representation learning for speech recognition, generation, and compression. The Hidden-Unit BERT (HuBERT) approach was proposed for self-supervised speech representation learning, which utilizes an offline clustering step to provide aligned target labels for a BERT-like prediction loss. This annotator is compatible with all the models trained/fine-tuned by using HubertForCTC for PyTorch or TFHubertForCTC for TensorFlow models in HuggingFace πŸ€—

image

HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Units by Wei-Ning Hsu, Benjamin Bolte, Yao-Hung Hubert Tsai, Kushal Lakhotia, Ruslan Salakhutdinov, Abdelrahman Mohamed.

Swin Transformer

NEW: Introducing SwinForImageClassification annotator in Spark NLP πŸš€. SwinForImageClassification can load transformer-based deep learning models with state-of-the-art performance in vision tasks. Swin Transformer precedes Vision Transformer (ViT) (Dosovitskiy et al., 2020) with great accuracy and efficiency. This annotator is compatible with all the models trained/fine-tuned by using SwinForImageClassification for PyTorch or TFSwinForImageClassification for TensorFlow models in HuggingFace πŸ€—

image

Swin Transformer: Hierarchical Vision Transformer using Shifted Windows by Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, Baining Guo.

Zero-Shot for Named Entity Recognition

Zero-Shot Learning refers to the process by which a model learns how to recognize objects (image, text, any features) without any labeled training data to help in the classification.

NEW: Introducing ZeroShotNerModel annotator in Spark NLP πŸš€. You can use the ZeroShotNerModel annotator to construct simple questions/answers mapped to NER labels like PERSON, NORP and etc. We use RoBERTa for Question Answering architecture behind the hood and this allows you to use any of the 460 models available on Models Hub to build your Zero-shot Entity Recognition with zero training dataset!

zero_shot_ner = ZeroShotNerModel.pretrained("roberta_base_qa_squad2", "en") \
    .setEntityDefinitions(
        {
            "NAME": ["What is his name?", "What is my name?", "What is her name?"],
            "CITY": ["Which city?", "Which is the city?"]
        }) \
    .setInputCols(["sentence", "token"]) \
    .setOutputCol("zero_shot_ner")

This powerful annotator with such simple rules can detect those entities from the following input: "My name is Clara, I live in New York and Hellen lives in Paris."

+-----------------------------------------------------------------+------+------+----------+------------------+
    |result                                                           |result|word  |confidence|question          |
    +-----------------------------------------------------------------+------+------+----------+------------------+
    |[My name is Clara, I live in New York and Hellen lives in Paris.]|B-CITY|Paris |0.5328949 |Which is the city?|
    |[My name is Clara, I live in New York and Hellen lives in Paris.]|B-NAME|Clara |0.9360068 |What is my name?  |
    |[My name is Clara, I live in New York and Hellen lives in Paris.]|B-CITY|New   |0.83294415|Which city?       |
    |[My name is Clara, I live in New York and Hellen lives in Paris.]|I-CITY|York  |0.83294415|Which city?       |
    |[My name is Clara, I live in New York and Hellen lives in Paris.]|B-NAME|Hellen|0.45366877|What is her name? |
    +-----------------------------------------------------------------+------+------+----------+------------------+

CamemBERT for Question Answering

NEW: Introducing CamemBertForQuestionAnswering annotator in Spark NLP πŸš€. CamemBertForQuestionAnswering can load CamemBERT Models with a span classification head on top for extractive question-answering tasks like SQuAD (linear layers on top of the hidden-states output to compute span start logits and span end logits). This annotator is compatible with all the models trained/fine-tuned by using CamembertForQuestionAnswering for PyTorch or TFCamembertForQuestionAnswering for TensorFlow in HuggingFace πŸ€—

Models Hub

Introduces a new filter by annotator which should help to navigate and find models easier:

image

β­πŸ› Improvements & Bug Fixes

  • New Date2Chunk annotator to convert DATE outputs coming from DateMatcher and MultiDateMatcher annotators to CHUNK that is acceptable by a wider range of annotators
  • Spark NLP 4.3.0 supports Apple Silicon M1 and M2 (still under experimental status until GitHub supports Apple Silicon officially). We have refactored the name m1 to silicon and apple_silicon in our code for better clarity
  • Add new templates for issues, docs, and feature requests on GitHub
  • Add a new log4j2 properties for Spark 3.3.x coming with Log4j 2.x to control the logs on Apache Spark
  • Cross compatibility for all saved pipelines for all major releases of Apache Spark and PySpark
  • Relocating Spark NLP examples to the examples directory in our main repository. We will update them on each release, will keep a history of the changes for each version, adding more languages, especially more use cases with Java and Scala
  • Add PyDoc documentation for ResourceDownloader in Python (clearCache(), showPublicModels(), showPublicPipelines(), and showAvailableAnnotators() )
  • Fix calculating delimiter id in CamemBERT annotators. The delimiter id is actually correct and doesn't need any offset
  • Fix AnalysisException exception that requires a different caught message for Spark 3.3
  • Fix copying existing models & pipelines on S3 before unzipping when cache_pretrained is defined as S3 bucket
  • Fix copying existing models & pipelines on GCP before unzipping when cache_pretrained is defined as GCP bucket
  • Fix loadSavedModel() trying to load external models for private buckets on S3 with better error handling and warnings
  • Enable the params argument in the Spark NLP start function. You can create a params = {} with all Spark NLP and Apache Spark configs and pass it when starting the Spark NLP session
  • Add support for doc id in CoNLL() class when trying to read CoNLL files with id inside each document's header
  • Welcoming 6 new Databricks runtimes to our Spark NLP family:
    • Databricks 12.0
    • Databricks 12.0 ML
    • Databricks 12.0 ML GPU
    • Databricks 12.1
    • Databricks 12.1 ML
    • Databricks 12.1 ML GPU
  • Welcoming 2 new EMR 6.x series to our Spark NLP family:
    • EMR 6.8.0 (Apache Spark 3.3.0 / Hadoop 3.2.1)
    • EMR 6.9.0 (Apache Spark 3.3.0 / Hadoop 3.3.3)
  • New article for semantic similarity with Spark NLP on Play/API/Swagger/ https://medium.com/spark-nlp/semantic-similarity-with-sparknlp-da148fafa3d8

Dependencies & Code Changes

  • Update Apache Spark 3.3.1 (not shipped with Spark NLP
  • Update GCP to 2.16.0
  • Update Scala test to 3.2.14
  • Start publishing spark-nlp-m1 Maven package as spark-nlp-silicon
  • Rename all read model traits to a generic name. A new ai module paving a path to another DL engine
  • Rename TF backends to more generic DL names
  • Refactor more duplicate codes in transformer embeddings

πŸ’Ύ Models

Spark NLP 4.3.0 comes with 1000+ state-of-the-art pre-trained transformer models in many languages.

Featured Models

Model Name Lang
DistilBertForQuestionAnswering distilbert_qa_en_de_vi_zh_es_model xx
DistilBertForQuestionAnswering distilbert_qa_extractive en
DistilBertForQuestionAnswering distilbert_qa_base_cased_squadv2 xx
RoBertaForQuestionAnswering [roberta_qa_roberta](https://nlp.johnsnowlabs.com/2023/01...
Read more

Contributors

agsfer, maziyarpanahi, and 11 other contributors
Assets 2
Loading
0 Join discussion

Spark NLP 4.2.8: Patch release

24 Jan 18:22
@maziyarpanahi maziyarpanahi
4.2.8
04b5ac3
Compare
Choose a tag to compare
Spark NLP 4.2.8: Patch release

πŸ“’ Overview

Spark NLP 4.2.8 πŸš€ comes with some important bug fixes and improvements. As a result, we highly recommend to update to this latest version if you are using Spark NLP 4.2.x.

As always, we would like to thank our community for their feedback, questions, and feature requests. πŸŽ‰

⭐ πŸ› Bug Fixes & Improvements

  • Fix the issue with optional keys (labels) in metadata when using XXXForSequenceClassitication annotators. This fixes Some(neg) -> 0.13602075 as neg -> 0.13602075 to be in harmony with all the other classifiers. #13396

before 4.2.8:

+-----------------------------------------------------------------------------------------------+
|label                                                                                          |
+-----------------------------------------------------------------------------------------------+
|[{category, 0, 87, pos, {sentence -> 0, Some(neg) -> 0.13602075, Some(pos) -> 0.8639792}, []}] |
|[{category, 0, 47, neg, {sentence -> 0, Some(neg) -> 0.7505674, Some(pos) -> 0.24943262}, []}] |
|[{category, 0, 17, pos, {sentence -> 0, Some(neg) -> 0.31065974, Some(pos) -> 0.6893403}, []}] |
|[{category, 0, 71, neg, {sentence -> 0, Some(neg) -> 0.5079189, Some(pos) -> 0.4920811}, []}]  |
+-----------------------------------------------------------------------------------------------+

after 4.2.8:

+-----------------------------------------------------------------------------------+
|label                                                                              |
+-----------------------------------------------------------------------------------+
|[{category, 0, 87, pos, {sentence -> 0, neg -> 0.13602075, pos -> 0.8639792}, []}] |
|[{category, 0, 47, neg, {sentence -> 0, neg -> 0.7505674, pos -> 0.24943262}, []}] |
|[{category, 0, 17, pos, {sentence -> 0, neg -> 0.31065974, pos -> 0.6893403}, []}] |
|[{category, 0, 71, neg, {sentence -> 0, neg -> 0.5079189, pos -> 0.4920811}, []}]  |
+-----------------------------------------------------------------------------------+
  • Introducing a config to skip LightPipeline validation for inputCols on the Python side for projects depending on Spark NLP. This toggle should only be used for specific annotators that do not follow the convention of predefined inputAnnotatorTypes and outputAnnotatorType #13402

πŸ“– Documentation

Installation

Python

#PyPI

pip install spark-nlp==4.2.8

Spark Packages

spark-nlp on Apache Spark 3.0.x, 3.1.x, 3.2.x, and 3.3.x (Scala 2.12):

spark-shell --packages com.johnsnowlabs.nlp:spark-nlp_2.12:4.2.8

pyspark --packages com.johnsnowlabs.nlp:spark-nlp_2.12:4.2.8

GPU

spark-shell --packages com.johnsnowlabs.nlp:spark-nlp-gpu_2.12:4.2.8

pyspark --packages com.johnsnowlabs.nlp:spark-nlp-gpu_2.12:4.2.8

M1

spark-shell --packages com.johnsnowlabs.nlp:spark-nlp-m1_2.12:4.2.8

pyspark --packages com.johnsnowlabs.nlp:spark-nlp-m1_2.12:4.2.8

AArch64

spark-shell --packages com.johnsnowlabs.nlp:spark-nlp-aarch64_2.12:4.2.8

pyspark --packages com.johnsnowlabs.nlp:spark-nlp-aarch64_2.12:4.2.8

Maven

spark-nlp on Apache Spark 3.0.x, 3.1.x, 3.2.x, and 3.3.x:

<dependency>
    <groupId>com.johnsnowlabs.nlp</groupId>
    <artifactId>spark-nlp_2.12</artifactId>
    <version>4.2.8</version>
</dependency>

spark-nlp-gpu:

<dependency>
    <groupId>com.johnsnowlabs.nlp</groupId>
    <artifactId>spark-nlp-gpu_2.12</artifactId>
    <version>4.2.8</version>
</dependency>

spark-nlp-m1:

<dependency>
    <groupId>com.johnsnowlabs.nlp</groupId>
    <artifactId>spark-nlp-m1_2.12</artifactId>
    <version>4.2.8</version>
</dependency>

spark-nlp-aarch64:

<!-- https://mvnrepository.com/artifact/com.johnsnowlabs.nlp/spark-nlp-aarch64 -->
<dependency>
    <groupId>com.johnsnowlabs.nlp</groupId>
    <artifactId>spark-nlp-aarch64_2.12</artifactId>
    <version>4.2.8</version>
</dependency>

FAT JARs

What's Changed

Full Changelog: 4.2.7...4.2.8

Contributors

pabla, albertoandreottiATgmail, and 9 other contributors
Assets 2
Loading
0 Join discussion

Spark NLP 4.2.7: Patch release

12 Jan 16:46
@maziyarpanahi maziyarpanahi
4.2.7
b87e14d
Compare
Choose a tag to compare
Spark NLP 4.2.7: Patch release

πŸ“’ Overview

Spark NLP 4.2.7 πŸš€ comes with some important bug fixes and improvements. As a result, we highly recommend to update to this latest version if you are using Spark NLP 4.2.x.

As always, we would like to thank our community for their feedback, questions, and feature requests. πŸŽ‰

πŸ› ⭐ Bug Fixes & Enhancements

  • Fix outputAnnotatorType issue in pipelines with Finisher annotator. This change adds outputAnnotatorType to AnnotatorTransformer to avoid loading outputAnnotatorType attribute when a stage in pipeline does not use it.
  • Fix the wrong sentence index calculation in metadata by annotators in the pipeline when setExplodeSentences param was set to true in SentenceDetector annotator
  • Fix the issue in Tokenizer when a custom pattern is used with lookahead/-behinds and it has 0 width matches. This led to indexes not being calculated correctly
  • Fix missing to output embeddings in .fullAnnotate() method when parseEmbeddings param was set to True/true
  • Fix broken links to the Python API pages, as the generation of the PyDocs was slightly changed in a previous release. This makes the Python APIs accessible from the Annotators and Transformers pages like before
  • Change default values of explodeEntities and mergeEntities parameters to true in GraphExctraction annotator
  • Better error handling when there are empty paths/relations in GraphExctractionannotator. New message will better guide the user on how to configure GraphExtraction to output meaningful relationships
  • Removed the duplicated definition of method setWeightedDistPath from ContextSpellCheckerApproach

πŸ“– Documentation

Installation

Python

#PyPI

pip install spark-nlp==4.2.7

Spark Packages

spark-nlp on Apache Spark 3.0.x, 3.1.x, 3.2.x, and 3.3.x (Scala 2.12):

spark-shell --packages com.johnsnowlabs.nlp:spark-nlp_2.12:4.2.7

pyspark --packages com.johnsnowlabs.nlp:spark-nlp_2.12:4.2.7

GPU

spark-shell --packages com.johnsnowlabs.nlp:spark-nlp-gpu_2.12:4.2.7

pyspark --packages com.johnsnowlabs.nlp:spark-nlp-gpu_2.12:4.2.7

M1

spark-shell --packages com.johnsnowlabs.nlp:spark-nlp-m1_2.12:4.2.7

pyspark --packages com.johnsnowlabs.nlp:spark-nlp-m1_2.12:4.2.7

AArch64

spark-shell --packages com.johnsnowlabs.nlp:spark-nlp-aarch64_2.12:4.2.7

pyspark --packages com.johnsnowlabs.nlp:spark-nlp-aarch64_2.12:4.2.7

Maven

spark-nlp on Apache Spark 3.0.x, 3.1.x, 3.2.x, and 3.3.x:

<dependency>
    <groupId>com.johnsnowlabs.nlp</groupId>
    <artifactId>spark-nlp_2.12</artifactId>
    <version>4.2.7</version>
</dependency>

spark-nlp-gpu:

<dependency>
    <groupId>com.johnsnowlabs.nlp</groupId>
    <artifactId>spark-nlp-gpu_2.12</artifactId>
    <version>4.2.7</version>
</dependency>

spark-nlp-m1:

<dependency>
    <groupId>com.johnsnowlabs.nlp</groupId>
    <artifactId>spark-nlp-m1_2.12</artifactId>
    <version>4.2.7</version>
</dependency>

spark-nlp-aarch64:

<!-- https://mvnrepository.com/artifact/com.johnsnowlabs.nlp/spark-nlp-aarch64 -->
<dependency>
    <groupId>com.johnsnowlabs.nlp</groupId>
    <artifactId>spark-nlp-aarch64_2.12</artifactId>
    <version>4.2.7</version>
</dependency>

FAT JARs

What's Changed

@dcecchini @Cabir40 @agsfer @gadde5300 @bunyamin-polat @rpranab @jdobes-cz @josejuanmartinez @diatrambitas @maziyarpanahi

Full Changelog: 4.2.6...4.2.7

Contributors

agsfer, maziyarpanahi, and 8 other contributors
Assets 2
Loading
0 Join discussion

Spark NLP 4.2.6: Patch release

21 Dec 09:54
@maziyarpanahi maziyarpanahi
4.2.6
4a91ba3
Compare
Choose a tag to compare
Spark NLP 4.2.6: Patch release

⭐ Improvements

  • Updating Spark & PySpark dependencies from 3.2.1 to 3.2.3 in provided scripts and in all the documentation

πŸ› Bug Fixes

  • Fix the broken TypedDependencyParserApproach and TypedDependencyParserModel annotators used in Python (this bug was introduced in 4.2.5 release)
  • Fix the broken Python API documentation

πŸ“– Documentation

Installation

Python

#PyPI

pip install spark-nlp==4.2.6

Spark Packages

spark-nlp on Apache Spark 3.0.x, 3.1.x, 3.2.x, and 3.3.x (Scala 2.12):

spark-shell --packages com.johnsnowlabs.nlp:spark-nlp_2.12:4.2.6

pyspark --packages com.johnsnowlabs.nlp:spark-nlp_2.12:4.2.6

GPU

spark-shell --packages com.johnsnowlabs.nlp:spark-nlp-gpu_2.12:4.2.6

pyspark --packages com.johnsnowlabs.nlp:spark-nlp-gpu_2.12:4.2.6

M1

spark-shell --packages com.johnsnowlabs.nlp:spark-nlp-m1_2.12:4.2.6

pyspark --packages com.johnsnowlabs.nlp:spark-nlp-m1_2.12:4.2.6

AArch64

spark-shell --packages com.johnsnowlabs.nlp:spark-nlp-aarch64_2.12:4.2.6

pyspark --packages com.johnsnowlabs.nlp:spark-nlp-aarch64_2.12:4.2.6

Maven

spark-nlp on Apache Spark 3.0.x, 3.1.x, 3.2.x, and 3.3.x:

<dependency>
    <groupId>com.johnsnowlabs.nlp</groupId>
    <artifactId>spark-nlp_2.12</artifactId>
    <version>4.2.6</version>
</dependency>

spark-nlp-gpu:

<dependency>
    <groupId>com.johnsnowlabs.nlp</groupId>
    <artifactId>spark-nlp-gpu_2.12</artifactId>
    <version>4.2.6</version>
</dependency>

spark-nlp-m1:

<dependency>
    <groupId>com.johnsnowlabs.nlp</groupId>
    <artifactId>spark-nlp-m1_2.12</artifactId>
    <version>4.2.6</version>
</dependency>

spark-nlp-aarch64:

<!-- https://mvnrepository.com/artifact/com.johnsnowlabs.nlp/spark-nlp-aarch64 -->
<dependency>
    <groupId>com.johnsnowlabs.nlp</groupId>
    <artifactId>spark-nlp-aarch64_2.12</artifactId>
    <version>4.2.6</version>
</dependency>

FAT JARs

What's Changed

Contributors

'@gadde5300 @diatrambitas @Cabir40 @josejuanmartinez @danilojsl @jsl-builder @DevinTDHa @maziyarpanahi @dcecchini @agsfer '

Full Changelog: 4.2.5...4.2.6

Contributors

agsfer, maziyarpanahi, and 8 other contributors
Assets 2
Loading

Spark NLP 4.2.5: New CamemBERT for sequence classification, better pipeline validation in LightPipeline, new Databricks 11.3 runtime, new EMR 6.8/6.9 versions with Spark 3.3, updated notebooks with latest TensorFlow 2.11, 400+ state-of-the-art models and many more!

16 Dec 09:03
@maziyarpanahi maziyarpanahi
4.2.5
64b333f
Compare
Choose a tag to compare
Spark NLP 4.2.5: New CamemBERT for sequence classification, better pipeline validation in LightPipeline, new Databricks 11.3 runtime, new EMR 6.8/6.9 versions with Spark 3.3, updated notebooks with latest TensorFlow 2.11, 400+ state-of-the-art models and many more!

πŸ“’ Overview

Spark NLP 4.2.5 πŸš€ comes with a new CamemBERT for sequence classification annotator (multi-class & multi-label), new pipeline validation for LightPipeline in Python, 26 updated noteooks to use the latest TensorFlow and Transformers libraries, support for new Databricks 11.3 runtime, support for new EMR versions of 6.8 and 6.9 (only EMR versions with Spark 3.3), over 400+ state-of-the-art multi-lingual pretrained models, and bug fixes.

Do not forget to visit Models Hub with over 11700+ free and open-source models & pipelines. As always, we would like to thank our community for their feedback, questions, and feature requests. πŸŽ‰

⭐ New Features & improvements

  • NEW: Introducing CamemBertForSequenceClassification annotator in Spark NLP πŸš€. CamemBertForSequenceClassification can load CamemBERT Models with sequence classification/regression head on top (a linear layer on top of the pooled output) e.g. for multi-class document classification tasks. This annotator is compatible with all the models trained/fine-tuned by using CamembertForSequenceClassification for PyTorch or TFCamembertForSequenceClassification for TensorFlow in HuggingFace πŸ€—
  • NEW: Add AnnotatorType validation in Spark NLP LightPipeline. Currently, a misconfiguration of inputCols in an annotator in a pipeline raises an exception when using transform method, but in LightPipeline it only outputs empty values. This behavior can confuse users, this change introduces a validation that will raise an exception now in LightPipeline too.
    • Add outputAnnotatorType for all annotators in Python
    • Add inputAnnotatorTypes and outputAnnotatorType requirement validation for all subclasses derived from AnnotatorApproach and AnnotatorModel
    • Adding AnnotatorType validation in LightPipeline
  • NEW: Migrate 26 notenooks to import external Transformer models into Spark NLP. These notebooks now come with latest TensorFlow 2.11.0 and HuggingFace 4.25.1 releases. The notebooks also have TF signatures with data input types explicitly set to guarantee model sanity once imported into Spark NLP
  • Add validation for the number and type of columns set in TFNerDLGraphBuilder annotator. In efforts to avoid wrong definition of columns when using Spark NLP annotators in Python
  • Add more details to Alphabet error message in EntityRuler annotator to better guide users
  • Add instructions on how to resolve RocksDB incompatibilities when using Spark NLP with an M1 machine
  • Welcoming new Databricks runtimes support
    • 11.3
    • 11.3 ML
    • 11.3 GPU
  • Welcoming new EMR versions support
    • 6.8.0
    • 6.9.0
  • Refactor and implement a better error handling in ResourceDownloader. This change removes getObjectFromS3 allowing AWS SDK to rise the correspondent error. In addition, this change also refactors ResourceDownloader to reflect the intention of each credential type on the downloader
  • Implement full build and test of all unit tests base on Apache Spark 3.0.x, 3.1.x, 3.2.x, and 3.3.x major releases
  • UpdateUpgrade sbt-assembly to 1.2.0 that comes with lots of performance improvements. This benefits those who are trying to package Spark NLP as a Fat JAR
  • Update sbt to 1.8.0 with improvements and bug fixes, but mostly for CVEs fixes:
  • Use the new withIncludeScala in assemblyOption instead of value

πŸ› Bug Fixes

  • Fix an issue with the BigTextMatcher Annotator, where it would not match entities with overlapping definitions. For Example, if both lung and lung cancer are defined, lung would not be matched in a given text. This was due to an abstraction error of one of the subclasses of the BigTextMatcher during construction of the underlying data structure
  • Fix indexing issue for RegexTokenizer annotator. If the document was split into sentences, the index of the sentence inside the document was not taken into consideration for the indexes of the tokens. This would lead to further issues down the pipeline, where tokens would be filtered while unpacking them for other Annotators
  • Refactor the Resolvers object in Spark NLP's dependency to avoid the conflict with the Resolvers inside the new sbt

πŸ›‘ Known Issues

  • TypedDependencyParserModel annotator fails in Python in this release (will be fixed in 4.2.6 release next week)

Models

Spark NLP 4.2.5 comes with 400+ state-of-the-art pre-trained transformer models in many languages.

Featured Models

Model Name Lang
RoBertaForSequenceClassification roberta_classifier_autotrain_neurips_chanllenge_1287149282 en
RoBertaForSequenceClassification roberta_classifier_autonlp_imdb_rating_625417974 en
RoBertaForSequenceClassification RoBertaForSequenceClassification bn
RoBertaForSequenceClassification roberta_classifier_autotrain_citizen_nlu_hindi_1370952776 hi
RoBertaForSequenceClassification roberta_classifier_detect_acoso_twitter es
RoBertaForQuestionAnswering roberta_qa_deepset_base_squad2 en
RoBertaForQuestionAnswering roberta_qa_icebert is
RoBertaForQuestionAnswering roberta_qa_mrm8488_base_bne_finetuned_s_c es
RoBertaForQuestionAnswering roberta_qa_base_bne_squad2 es
BertEmbeddings bert_embeddings_rbt3 zh
BertEmbeddings bert_embeddings_base_it_cased it
BertEmbeddings bert_embeddings_base_indonesian_522m id
BertEmbeddings bert_embeddings_base_german_uncased `de
BertEmbeddings bert_embeddings_base_japanese_char ja
BertEmbeddings bert_embeddings_bangla_base bn
BertEmbeddings bert_embeddings_base_arabertv01 ar

Spark NLP covers the following languages:

English ,Multilingual ,Afrikaans ,Afro-Asiatic languages ,Albanian ,Altaic languages ,American Sign Language ,Amharic ,Arabic ,Argentine Sign Language ,Armenian ,Artificial languages ,Atlantic-Congo languages ,Austro-Asiatic languages ,Austronesian languages ,Azerbaijani ,Baltic languages ,Bantu languages ,Basque ,Basque (family) ,Belarusian ,Bemba (Zambia) ,Bengali, Bangla ,Berber languages ,Bihari ,Bislama ,Bosnian ,Brazilian Sign Language ,Breton ,Bulgarian ,Catalan ,Caucasian languages ,Cebuano ,Celtic languages ,Central Bikol ,Chichewa, Chewa, Nyanja ,Chilean Sign Language ,Chinese ,Chuukese ,Colombian Sign Language ,Congo Swahili ,Croatian ,Cushitic languages ,Czech ,Danish ,Dholuo, Luo (Kenya and Tanzania) ,Dravidian languages ,Dutch ,East Slavic languages ,Eastern Malayo-Polynesian languages ,Efik ,Esperanto ,Estonian ,Ewe ,Fijian ,Finnish ,Finnish Sign Language ,Finno-Ugrian languages ,French ,French-based creoles and pidgins ,Ga ,Galician ,Ganda ,Georgian ,German ,Germanic languages ,Gilbertese ,Greek (modern) ,Greek languages ,Gujarati ,Gun ,Haitian, Haitian Creole ,Hausa ,Hebrew (modern) ,Hiligaynon ,Hindi ,Hiri Motu ,Hungarian ,Icelandic ,Igbo ,Iloko ,Indic languages ,Indo-European languages ,Indo-Iranian languages ,Indonesian ,Irish ,Isoko ,Isthmus Zapotec ,Italian ,Italic languages ,Japanese ,Japanese ,Kabyle ,Kalaallisut, Greenlandic ,Kannada ,Kaonde ,Kinyarwanda ,Kirundi ,Kongo ,Korean ,Kwangali ,Kwanyama, Kuanyama ,Latin ,Latvian ,Lingala ,Lithuanian ,Louisiana Creole ,Lozi ,Luba-Katanga ,Luba-Lulua ,Lunda ,Lushai ,Luvale ,Macedonian ,Malagasy ,Malay ,Malayalam ,Malayo-Polynesian languages ,Maltese ,Manx ,Marathi (Marāṭhī) ,Marshallese ,Mexican Sign Language ,Mon-Khmer languages ,Morisyen ,Mossi ,Multiple languages ,Ndonga ,Nepali ,Niger-Kordofanian languages ,`N...

Read more

Contributors

agsfer, maziyarpanahi, and 9 other contributors
Assets 2
Loading
3 Join discussion

Spark NLP 4.2.4: Introducing support for GCP storage for pre-trained models, update to TensorFlow 2.7.4 with CVEs fixes, improvements, and bug fixes

28 Nov 19:01
@maziyarpanahi maziyarpanahi
4.2.4
b55db41
Compare
Choose a tag to compare
Spark NLP 4.2.4: Introducing support for GCP storage for pre-trained models, update to TensorFlow 2.7.4 with CVEs fixes, improvements, and bug fixes

πŸ“’ Overview

Spark NLP 4.2.4 πŸš€ comes with new support for GCP storage to automatically download and load models & pipelines via setting the cache_pretrained path, update to TensorFlow 2.7.4 with security patch fixes, lots of improvements in our documentation, improvements, and bug fixes.

Do not forget to visit Models Hub with over 11400+ free and open-source models & pipelines. As always, we would like to thank our community for their feedback, questions, and feature requests. πŸŽ‰

⭐ New Features & improvements

  • Introducing support for GCP storage to automatically download and load pre-trained models/pipelines from cache_pretrained directory
  • Update to TensorFlow 2.7.4 with bug and CVEs fixes. Details about bugs and CVEs fixes: 417e2a1
  • Improve error handling while importing external TensorFlow models into Spark NLP
  • Improve error messages when importing external models from remote storages like DBFS, S3, and HDFS
  • Update documentation on how to use testDataset param in NerDLApproach, ClassifierDLApproach, MultiClassifierDLApproach, and SentimentDLApproach
  • Update installation instructions for the Apple M1 chip
  • Add support for future decoder-encoder models with 2 separated models

πŸ› Bug Fixes

  • Add missing setPreservePosition in NerConverter
  • Add missing inputAnnotatorTypes to BigTextMatcher, ViveknSentimentModel, and NerConverter annotators
  • Fix all wrong example codes provided for LemmatizerModel in Models Hub
  • Fix the t5_grammar_error_corrector model to be compatible with Spark NLP 4.0+
  • Fix provided notebook to import Longformer models from Hugging Face into Spark NLP

πŸ““ New Notebooks

Spark NLP Notebooks Colab
Spark NLP Conf Dowbload and Load Model from GCP Storage Open In Colab
LongformerEmbeddings HuggingFace in Spark NLP - Longformer Open In Colab

πŸ“– Documentation

Installation

Python

#PyPI

pip install spark-nlp==4.2.4

Spark Packages

spark-nlp on Apache Spark 3.0.x, 3.1.x, 3.2.x, and 3.3.x (Scala 2.12):

spark-shell --packages com.johnsnowlabs.nlp:spark-nlp_2.12:4.2.4

pyspark --packages com.johnsnowlabs.nlp:spark-nlp_2.12:4.2.4

GPU

spark-shell --packages com.johnsnowlabs.nlp:spark-nlp-gpu_2.12:4.2.4

pyspark --packages com.johnsnowlabs.nlp:spark-nlp-gpu_2.12:4.2.4

M1

spark-shell --packages com.johnsnowlabs.nlp:spark-nlp-m1_2.12:4.2.4

pyspark --packages com.johnsnowlabs.nlp:spark-nlp-m1_2.12:4.2.4

AArch64

spark-shell --packages com.johnsnowlabs.nlp:spark-nlp-aarch64_2.12:4.2.4

pyspark --packages com.johnsnowlabs.nlp:spark-nlp-aarch64_2.12:4.2.4

Maven

spark-nlp on Apache Spark 3.0.x, 3.1.x, 3.2.x, and 3.3.x:

<dependency>
    <groupId>com.johnsnowlabs.nlp</groupId>
    <artifactId>spark-nlp_2.12</artifactId>
    <version>4.2.4</version>
</dependency>

spark-nlp-gpu:

<dependency>
    <groupId>com.johnsnowlabs.nlp</groupId>
    <artifactId>spark-nlp-gpu_2.12</artifactId>
    <version>4.2.4</version>
</dependency>

spark-nlp-m1:

<dependency>
    <groupId>com.johnsnowlabs.nlp</groupId>
    <artifactId>spark-nlp-m1_2.12</artifactId>
    <version>4.2.4</version>
</dependency>

spark-nlp-aarch64:

<!-- https://mvnrepository.com/artifact/com.johnsnowlabs.nlp/spark-nlp-aarch64 -->
<dependency>
    <groupId>com.johnsnowlabs.nlp</groupId>
    <artifactId>spark-nlp-aarch64_2.12</artifactId>
    <version>4.2.4</version>
</dependency>

FAT JARs

What's Changed

Full Changelog: 4.2.3...4.2.4

Contributors

pabla, agsfer, and 11 other contributors
Assets 2
Loading
0 Join discussion

Spark NLP 4.2.1: Over 230 state-of-the-art Transformer Vision (ViT) pretrained pipelines, new multi-lingual support for Word Segmentation, add LightPipeline support to Automatic Speech Recognition pipelines, support for processed audio files in type Double for Wav2Vec2, and bug fixes

28 Nov 18:14
@maziyarpanahi maziyarpanahi
4.2.1
b55db41
Compare
Choose a tag to compare
Spark NLP 4.2.1: Over 230 state-of-the-art Transformer Vision (ViT) pretrained pipelines, new multi-lingual support for Word Segmentation, add LightPipeline support to Automatic Speech Recognition pipelines, support for processed audio files in type Double for Wav2Vec2, and bug fixes

πŸ“’ Overview

Spark NLP 4.2.1 πŸš€ comes with a new multi-lingual support for Word Segmentation mostly used for (but not limited to) Chinese, Japanese, Korean, and so on, adding Automatic Speech Recognition (ASR) pipelines to LightPipeline arsenal for faster computation of smaller datasets without Apache Spark (e.g. RESTful API use case), adding support for processed audio files in type of Double in addition to Float for Wav2Vec2, over 230+ state-of-the-art Transformer Vision (ViT) pretrained pipelines for 1-line Image Classification, and bug fixes.

Do not forget to visit Models Hub with over 11400+ free and open-source models & pipelines. As always, we would like to thank our community for their feedback, questions, and feature requests. πŸŽ‰

⭐ New Features & improvements

  • NEW: Support for multi-lingual WordSegmenter. Add enableRegexTokenizer feature in WordSegmenter to support word segmentation within mixed and multi-lingual content #12854
  • NEW: Add support for Audio/ASR (Wav2Vec2) support to LightPipeline #12895
  • NEW: Add support for Double type in addition to Float type to AudioAssembler annotator #12904
  • Improve error handling in fullAnnotateImage for LightPipeline #12868
  • Add SpanBertCoref annotator to all docs #12889

Bug Fixes

  • Fix feeding fullAnnotate in Lightpipeline with a list that started to fail in 4.2.0 release
  • Fix exception in ContextSpellCheckerModel when updateVocabClass is used with append set to true #12875
  • Fix exception in Chunker annotator #12901

πŸ““ New Notebooks

Spark NLP Notebooks Colab
SpanBertCorefModel Coreference Resolution with SpanBertCorefModel Open In Colab
WordSegmenter Train and inference multi-lingual Word Segmenter Open In Colab

Models

Spark NLP 4.2.1 comes with 230+ state-of-the-art pre-trained Transformer Vision (ViT) pipeline:

Featured Pipelines

Pipeline Name Lang
PretrainedPipeline pipeline_image_classifier_vit_base_patch16_224_finetuned_eurosat en
PretrainedPipeline pipeline_image_classifier_vit_base_beans_demo_v5 en
PretrainedPipeline pipeline_image_classifier_vit_animal_classifier_huggingface en
PretrainedPipeline pipeline_image_classifier_vit_Infrastructures en
PretrainedPipeline pipeline_image_classifier_vit_blocks en
PretrainedPipeline pipeline_image_classifier_vit_beer_whisky_wine_detection en
PretrainedPipeline pipeline_image_classifier_vit_base_xray_pneumonia en
PretrainedPipeline pipeline_image_classifier_vit_baseball_stadium_foods en
PretrainedPipeline pipeline_image_classifier_vit_dog_vs_chicken en

Check 460+ Transformer Vision (ViT) models & pipelines for Models Hub - Image Classification

Spark NLP covers the following languages:

English ,Multilingual ,Afrikaans ,Afro-Asiatic languages ,Albanian ,Altaic languages ,American Sign Language ,Amharic ,Arabic ,Argentine Sign Language ,Armenian ,Artificial languages ,Atlantic-Congo languages ,Austro-Asiatic languages ,Austronesian languages ,Azerbaijani ,Baltic languages ,Bantu languages ,Basque ,Basque (family) ,Belarusian ,Bemba (Zambia) ,Bengali, Bangla ,Berber languages ,Bihari ,Bislama ,Bosnian ,Brazilian Sign Language ,Breton ,Bulgarian ,Catalan ,Caucasian languages ,Cebuano ,Celtic languages ,Central Bikol ,Chichewa, Chewa, Nyanja ,Chilean Sign Language ,Chinese ,Chuukese ,Colombian Sign Language ,Congo Swahili ,Croatian ,Cushitic languages ,Czech ,Danish ,Dholuo, Luo (Kenya and Tanzania) ,Dravidian languages ,Dutch ,East Slavic languages ,Eastern Malayo-Polynesian languages ,Efik ,Esperanto ,Estonian ,Ewe ,Fijian ,Finnish ,Finnish Sign Language ,Finno-Ugrian languages ,French ,French-based creoles and pidgins ,Ga ,Galician ,Ganda ,Georgian ,German ,Germanic languages ,Gilbertese ,Greek (modern) ,Greek languages ,Gujarati ,Gun ,Haitian, Haitian Creole ,Hausa ,Hebrew (modern) ,Hiligaynon ,Hindi ,Hiri Motu ,Hungarian ,Icelandic ,Igbo ,Iloko ,Indic languages ,Indo-European languages ,Indo-Iranian languages ,Indonesian ,Irish ,Isoko ,Isthmus Zapotec ,Italian ,Italic languages ,Japanese ,Japanese ,Kabyle ,Kalaallisut, Greenlandic ,Kannada ,Kaonde ,Kinyarwanda ,Kirundi ,Kongo ,Korean ,Kwangali ,Kwanyama, Kuanyama ,Latin ,Latvian ,Lingala ,Lithuanian ,Louisiana Creole ,Lozi ,Luba-Katanga ,Luba-Lulua ,Lunda ,Lushai ,Luvale ,Macedonian ,Malagasy ,Malay ,Malayalam ,Malayo-Polynesian languages ,Maltese ,Manx ,Marathi (MarāṭhΔ«) ,Marshallese ,Mexican Sign Language ,Mon-Khmer languages ,Morisyen ,Mossi ,Multiple languages ,Ndonga ,Nepali ,Niger-Kordofanian languages ,Nigerian Pidgin ,Niuean ,North Germanic languages ,Northern Sotho, Pedi, Sepedi ,Norwegian ,Norwegian BokmΓ₯l ,Norwegian Nynorsk ,Nyaneka ,Oromo ,Pangasinan ,Papiamento ,Persian (Farsi) ,Peruvian Sign Language ,Philippine languages ,Pijin ,Pohnpeian ,Polish ,Portuguese ,Portuguese-based creoles and pidgins ,Punjabi (Eastern) ,Romance languages ,Romanian ,Rundi ,Russian ,Ruund ,Salishan languages ,Samoan ,San Salvador Kongo ,Sango ,Semitic languages ,Serbo-Croatian ,Seselwa Creole French ,Shona ,Sindhi ,Sino-Tibetan languages ,Slavic languages ,Slovak ,Slovene ,Somali ,South Caucasian languages ,South Slavic languages ,Southern Sotho ,Spanish ,Spanish Sign Language ,Sranan Tongo ,Swahili ,Swati ,Swedish ,Tagalog ,Tahitian ,Tai ,Tamil ,Telugu ,Tetela ,Tetun Dili ,Thai ,Tigrinya ,Tiv ,Tok Pisin ,Tonga (Tonga Islands) ,Tonga (Zambia) ,Tsonga ,Tswana ,Tumbuka ,Turkic languages ,Turkish ,Tuvalu ,Tzotzil ,Ukrainian ,Umbundu ,Uralic languages ,Urdu ,Venda ,Venezuelan Sign Language ,Vietnamese ,Wallisian ,Walloon ,Waray (Philippines) ,Welsh ,West Germanic languages ,West Slavic languages ,Western Malayo-Polynesian languages ,Wolaitta, Wolaytta ,Wolof ,Xhosa ,Yapese ,Yiddish ,Yoruba ,Yucatec Maya, Yucateco ,Zande (individual language) ,Zulu

The complete list of all 11000+ models & pipelines in 230+ languages is available on Models Hub

πŸ“– Documentation

Read more

Contributors

agsfer, maziyarpanahi, and 12 other contributors
Assets 2
Loading
0 Join discussion

Spark NLP 4.2.3: Improved CoNLLGenerator annotator, new rules parameter in RegexMatcher, new IAnnotation feature for LightPipeline in Scala, and bug fixes

10 Nov 20:17
@maziyarpanahi maziyarpanahi
4.2.3
e41cf90
Compare
Choose a tag to compare
Spark NLP 4.2.3: Improved CoNLLGenerator annotator, new rules parameter in RegexMatcher, new IAnnotation feature for LightPipeline in Scala, and bug fixes

πŸ“’ Overview

Spark NLP 4.2.3 πŸš€ comes with new improvements to the CoNLLGenerator annotator, a new way to pass rules to the RegexMatcher annotator, unifying control over a number of columns in setInputCols between the Scala and Python, new documentation for our new IAnnotation feature for those who are using Spark NLP in Scala, and bug fixes.

Do not forget to visit Models Hub with over 11400+ free and open-source models & pipelines. As always, we would like to thank our community for their feedback, questions, and feature requests. πŸŽ‰

⭐ New Features & improvements

  • Adding metadata sentence key parameter in order to select which metadata field to use as a sentence for the CoNLLGenerator annotator
  • Include escaping in the CoNLLGenerator annotator when writing to CSV and preserve special char token
  • Add rules and delimiter parameters to RegexMatcher annotator to support string as input in addition to a file
regexMatcher = RegexMatcher() \
      .setRules(["\\d{4}\\/\\d\\d\\/\\d\\d,date", "\\d{2}\\/\\d\\d\\/\\d\\d,short_date"]) \
      .setDelimiter(",") \
      .setInputCols(["sentence"]) \
      .setOutputCol("regex") \
      .setStrategy("MATCH_ALL")
  • Implement a new control over a number of accepted columns in Python. This will sync the behavior between Scala and Python where the user sets more columns than allowed inside setInputCols while using Spark NLP in Python
  • Add documentation for the new IAnnotation feature for Scala users

Bug Fixes

  • Fix NotSerializableException when the WordEmbeddings annotator is used over the K8s cluster while setEnableInMemoryStorage is set to true
  • Fix a bug in the RegexTokenizer annotator when it outputs the wrong indexes if the pattern includes splits that are not followed by a space
  • Fix training module failing on EMR due to a bad Apache Spark version detection. The use of the following classes was fixed on EMR: CoNLL(), CoNLLU(), POS(), and PubTator()
  • Fix a bug in the CoNLLGenerator annotator where the token has non-int metadata
  • Fix the wrong SentencePiece model's name required for DeBertaForQuestionAnswering and DeBertaEmbeddings when importing models
  • Fix NaNs result in some ViTForImageClassification models/pipelines

πŸ““ New Notebooks

πŸ“– Documentation

Installation

Python

#PyPI

pip install spark-nlp==4.2.3

Spark Packages

spark-nlp on Apache Spark 3.0.x, 3.1.x, 3.2.x, and 3.3.x (Scala 2.12):

spark-shell --packages com.johnsnowlabs.nlp:spark-nlp_2.12:4.2.3

pyspark --packages com.johnsnowlabs.nlp:spark-nlp_2.12:4.2.3

GPU

spark-shell --packages com.johnsnowlabs.nlp:spark-nlp-gpu_2.12:4.2.3

pyspark --packages com.johnsnowlabs.nlp:spark-nlp-gpu_2.12:4.2.3

M1

spark-shell --packages com.johnsnowlabs.nlp:spark-nlp-m1_2.12:4.2.3

pyspark --packages com.johnsnowlabs.nlp:spark-nlp-m1_2.12:4.2.3

AArch64

spark-shell --packages com.johnsnowlabs.nlp:spark-nlp-aarch64_2.12:4.2.3

pyspark --packages com.johnsnowlabs.nlp:spark-nlp-aarch64_2.12:4.2.3

Maven

spark-nlp on Apache Spark 3.0.x, 3.1.x, 3.2.x, and 3.3.x:

<dependency>
    <groupId>com.johnsnowlabs.nlp</groupId>
    <artifactId>spark-nlp_2.12</artifactId>
    <version>4.2.3</version>
</dependency>

spark-nlp-gpu:

<dependency>
    <groupId>com.johnsnowlabs.nlp</groupId>
    <artifactId>spark-nlp-gpu_2.12</artifactId>
    <version>4.2.3</version>
</dependency>

spark-nlp-m1:

<dependency>
    <groupId>com.johnsnowlabs.nlp</groupId>
    <artifactId>spark-nlp-m1_2.12</artifactId>
    <version>4.2.3</version>
</dependency>

spark-nlp-aarch64:

<!-- https://mvnrepository.com/artifact/com.johnsnowlabs.nlp/spark-nlp-aarch64 -->
<dependency>
    <groupId>com.johnsnowlabs.nlp</groupId>
    <artifactId>spark-nlp-aarch64_2.12</artifactId>
    <version>4.2.3</version>
</dependency>

FAT JARs

What's Changed

Read more

Contributors

pabla, albertoandreottiATgmail, and 13 other contributors
Assets 2
Loading
2 Join discussion

Spark NLP 4.2.2: Support DBFS, HDFS, and S3 for importing external models, unifying LightPipeline APIs across supported languages for Image Classification, new fullAnnotateImage for Scala, new fullAnnotateImageJava for Java, support LightPipeline for QuestionAnswering pre-trained pipelines, and bug fixes

27 Oct 18:07
@maziyarpanahi maziyarpanahi
4.2.2
39f3c01
Compare
Choose a tag to compare
Spark NLP 4.2.2: Support DBFS, HDFS, and S3 for importing external models, unifying LightPipeline APIs across supported languages for Image Classification, new fullAnnotateImage for Scala, new fullAnnotateImageJava for Java, support LightPipeline for QuestionAnswering pre-trained pipelines, and bug fixes

πŸ“’ Overview

Spark NLP 4.2.2 πŸš€ comes with support for DBFS, HDFS, and S3 in addition to local file systems when you are importing external models from TF Hub and Hugging Face, unifying LightPipeline APIs across Scala, Java, and Python languages for Image Classification, the new fullAnnotateImage for Scala, the new fullAnnotateImageJava for Java, the support for LightPipeline for QuestionAnswering pre-trained pipelines, and bug fixes.

Do not forget to visit Models Hub with over 11400+ free and open-source models & pipelines. As always, we would like to thank our community for their feedback, questions, and feature requests. πŸŽ‰

⭐ New Features & improvements

  • Add support for importing TensorFlow SavedModel from remote storages like DBFS, S3, and HDFS. From this release, you can import models saved from TF Hub and HuggingFace on a remote storage
  • Add support for fullAnnotate in LightPipeline for the path of images in Scala
  • Add fullAnnotate method in PretrainedPipeline for Scala
  • Add fullAnnotateJava method in PretrainedPipeline for Java
  • Add fullAnnotateImage to PretrainedPipeline for Scala
  • Add fullAnnotateImageJava to PretrainedPipeline for Java
  • Add support for Question Answering in fullAnnotate method in PretrainedPipeline
  • Add Predicted Entities to all Vision Transformers (ViT) models and pipelines

Bug Fixes

  • Unify the annotatorType name in Python and Scala for Spark schema in Annotation, AnnotationImage, and AnnotationAudio
  • Fix missing indexes in the RecursiveTokenizer annotator affecting downstream NLP tasks in the pipeline

πŸ““ New Notebooks

Spark NLP Notebooks Colab
WordSegmenter Import External SavedModel From Remote Open In Colab

πŸ“– Documentation

Installation

Python

#PyPI

pip install spark-nlp==4.2.2

Spark Packages

spark-nlp on Apache Spark 3.0.x, 3.1.x, 3.2.x, and 3.3.x (Scala 2.12):

spark-shell --packages com.johnsnowlabs.nlp:spark-nlp_2.12:4.2.2

pyspark --packages com.johnsnowlabs.nlp:spark-nlp_2.12:4.2.2

GPU

spark-shell --packages com.johnsnowlabs.nlp:spark-nlp-gpu_2.12:4.2.2

pyspark --packages com.johnsnowlabs.nlp:spark-nlp-gpu_2.12:4.2.2

M1

spark-shell --packages com.johnsnowlabs.nlp:spark-nlp-m1_2.12:4.2.2

pyspark --packages com.johnsnowlabs.nlp:spark-nlp-m1_2.12:4.2.2

Maven

spark-nlp on Apache Spark 3.0.x, 3.1.x, 3.2.x, and 3.3.x:

<dependency>
    <groupId>com.johnsnowlabs.nlp</groupId>
    <artifactId>spark-nlp_2.12</artifactId>
    <version>4.2.2</version>
</dependency>

spark-nlp-gpu:

<dependency>
    <groupId>com.johnsnowlabs.nlp</groupId>
    <artifactId>spark-nlp-gpu_2.12</artifactId>
    <version>4.2.2</version>
</dependency>

spark-nlp-m1:

<dependency>
    <groupId>com.johnsnowlabs.nlp</groupId>
    <artifactId>spark-nlp-m1_2.12</artifactId>
    <version>4.2.2</version>
</dependency>

FAT JARs

What's Changed

Contributors

@galiph @agsfer @pabla @josejuanmartinez @Cabir40 @maziyarpanahi @Meryem1425 @danilojsl @jsl-builder @jsl-models @ahmedlone127 @DevinTDHa @jdobes-cz @Damla-Gurbaz @Mary-Sci

New Contributors

Full Changelog: 4.2.1...4.2.2

Contributors

pabla, agsfer, and 13 other contributors
Assets 2
Loading
0 Join discussion

Spark NLP 4.2.0: Wav2Vec2 for Automatic Speech Recognition (ASR), TAPAS for Table Question Answering, CamemBERT for Token Classification, new evaluation metrics for external datasets in all classifiers, much faster EntityRuler, over 3000+ state-of-the-art multi-lingual models & pipelines, and many more!

27 Sep 14:29
@maziyarpanahi maziyarpanahi
4.2.0
4119c2d
Compare
Choose a tag to compare
Spark NLP 4.2.0: Wav2Vec2 for Automatic Speech Recognition (ASR), TAPAS for Table Question Answering, CamemBERT for Token Classification, new evaluation metrics for external datasets in all classifiers, much faster EntityRuler, over 3000+ state-of-the-art multi-lingual models & pipelines, and many more!

πŸ“’ Overview

For the first time ever we are delighted to announce Automatic Speech Recognition (ASR) support in Spark NLP by using state-of-the-art Wav2Vec2 models at scale πŸš€. This release also comes with Table Question Answering by TAPAS, CamemBERT for Token Classification, support for an external test dataset during training of all classifiers, much faster EntityRuler, 3000+ state-of-the-art models, and other enhancements and bug fixes!

We are also celebrating crossing 11000+ free and open-source models & pipelines in our Models Hub. πŸŽ‰ As always, we would like to thank our community for their feedback, questions, and feature requests.

⭐ New Features & improvements

  • NEW: Introducing Wav2Vec2ForCTC annotator in Spark NLP πŸš€. Wav2Vec2ForCTC can load Wav2Vec2 models for the Automatic Speech Recognition (ASR) task. Wav2Vec2 is a multi-modal model, that combines speech and text. It's the first multi-modal model of its kind we welcome in Spark NLP. This annotator is compatible with all the models trained/fine-tuned by using Wav2Vec2ForCTC for PyTorch or TFWav2Vec2ForCTC for TensorFlow models in HuggingFace πŸ€— (#12767)

image

wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations

  • NEW: Introducing TapasForQuestionAnswering annotator in Spark NLP πŸš€. TapasForQuestionAnswering can load TAPAS Models with a cell selection head and optional aggregation head on top for question-answering tasks on tables (linear layers on top of the hidden-states output to compute logits and optional logits_aggregation), e.g. for SQA, WTQ or WikiSQL-supervised tasks. TAPAS is a BERT-based model specifically designed (and pre-trained) for answering questions about tabular data. This annotator is compatible with all the models trained/fine-tuned by using TapasForQuestionAnswering for PyTorch or TFTapasForQuestionAnswering for TensorFlow models in HuggingFace πŸ€—

image

TAPAS: Weakly Supervised Table Parsing via Pre-training

  • NEW: Introducing CamemBertForTokenClassification annotator in Spark NLP πŸš€. CamemBertForTokenClassification can load CamemBERT Models with a token classification head on top (a linear layer on top of the hidden-states output) e.g. for Named-Entity-Recognition (NER) tasks. This annotator is compatible with all the models trained/fine-tuned by using CamembertForTokenClassification for PyTorch or TFCamembertForTokenClassification for TensorFlow in HuggingFace πŸ€—
    (#12752)
  • Implementing setTestDataset to evaluate metrics on an external dataset during training of Text Classifiers in Spark NLP. This feature is similar to NerDLApproach where metrics are calculated on each Epoch and have been added to the following multi-class/multi-label text classifier annotators: ClassifierDLApproach, SentimentDLApproach, and MultiClassifierDLApproach (#12796)
  • Refactoring and improving EntityRuler annotator inference to up to 24x faster especially when used with a long list of labels/entities. We speed up the inference process by implementing the Aho-Corasick algorithm to match patterns in a string. This requires the following changes when using EntityRuler #12634
  • Add support for S3 storage in the cache_folder where models are downloaded, extracted, and loaded from. Previously, we only supported all local file systems, HDFS, and DBFS. This new feature is especially useful for users on Kubernetes clusters with no access to HDFS or any other distributed file systems (#12707)
  • Implementing lookaround functionalities in DocumentNormalizer annotator. Currently, DocumentNormalizer has both lookahead and lookbehind functionalities. To extend support for more complex normalizations, especially within the clinical text we are introducing the lookaround feature (#12735)
  • Implementing setReplaceEntities param to NerOverwriter annotator to replace all the NER labels (entities) with the given new labels (entities) (#12745)

Bug Fixes

  • Fix a bug in generating the NerDL graph by using TF v2. The previous graph generated by the TFGraphBuilder annotator resulted in an exception when the length of the sequence was 1. This issue has been resolved and the new graphs created by TFGraphBuilder won't have this issue anymore (#12636)
  • Fix a bug introduced in the 4.0.0 release between Transformer-based Word Embeddings annotators. In the 4.0.0 release, the following annotators were migrated to BatchAnnotate to improve their performance, especially on GPU. However, a bug was introduced in sentence indices which when it is combined with SentenceEmbeddings for Text Classifications tasks (ClassifierDLApproach, SentimentDLApproach, and ClassifierDLApproach) resulted in low accuracy: AlbertEmbeddings, CamemBertEmbeddings, DeBertaEmbeddings, DistilBertEmbeddings, LongformerEmbeddings, RoBertaEmbeddings, XlmRoBertaEmbeddings, and XlnetEmbeddings (#12641)
  • Add support for a list of questions and context in LightPipline. Previously, only one context and question at a time were supported in LightPipeline for Question Answering annotators. We have added support to fullAnnotate and annotate to receive two lists of questions and contexts (#12653)
  • Fix division by zero exception in the GPT2Transformer annotator when the setDoSample param was set to true (#12661)
  • Fix AttributeError when PretrainedPipeline is used in Python with ImageAssembler as one of the stages (#12813)

πŸ““ New Notebooks

Spark NLP Notebooks Colab
Wav2Vec2ForCTC Automatic Speech Recognition in Spark NLP Open In Colab
ViTForImageClassification HuggingFace in Spark NLP - ViTForImageClassification Open In Colab
CamemBertForTokenClassification HuggingFace in Spark NLP - CamemBertForTokenClassification Open In Colab
ClassifierDLApproach ClassifierDL Train and Evaluate Open In Colab
MultiClassifierDLApproach MultiClassifierDL Train and Evaluate Open In Colab
SentimentDLApproach SentimentDL Train and Evaluate Open In Colab
Pretrained/cache_folder Download & Load Models From S3 Open In Colab
EntityRuler [EntityRuler](https://github.com/JohnSnowLabs/spark-nlp-workshop/...
Read more

Contributors

pabla, xusliebana, and 16 other contributors
Assets 2
Loading
0 Join discussion