From 64b333f7897ea6c7706ed8a6af976d88ffdb1c78 Mon Sep 17 00:00:00 2001 From: Maziyar Panahi Date: Thu, 15 Dec 2022 21:11:44 +0100 Subject: [PATCH] Adding new Databricks and EMR supported versions --- README.md | 3 +++ docs/en/transformers.md | 14 ++++++++------ python/README.md | 3 +++ 3 files changed, 14 insertions(+), 6 deletions(-) diff --git a/README.md b/README.md index 6c16f6915b7db7..60117a2d35ed0f 100644 --- a/README.md +++ b/README.md @@ -271,6 +271,8 @@ Spark NLP 4.2.5 has been tested and is compatible with the following runtimes: - 11.1 ML - 11.2 - 11.2 ML +- 11.3 +- 11.3 ML **GPU:** @@ -283,6 +285,7 @@ Spark NLP 4.2.5 has been tested and is compatible with the following runtimes: - 11.0 ML & GPU - 11.1 ML & GPU - 11.2 ML & GPU +- 11.3 ML & GPU NOTE: Spark NLP 4.0.x is based on TensorFlow 2.7.x which is compatible with CUDA11 and cuDNN 8.0.2. The only Databricks runtimes supporting CUDA 11 are 9.x and above as listed under GPU. diff --git a/docs/en/transformers.md b/docs/en/transformers.md index 9c7a1c6b74eca6..67daee9a2aca2f 100644 --- a/docs/en/transformers.md +++ b/docs/en/transformers.md @@ -27,7 +27,7 @@ sidebar: ### Overview -We have extended support for `HuggingFace` 🤗 and `TF Hub` exported models since `3.1.0` to equivalent Spark NLP 🚀 annotators. Starting this release, you can easily use the `saved_model` feature in HuggingFace within a few lines of codes and import any `BERT`, `DistilBERT`, `CamemBERT`, `RoBERTa`, `DeBERTa`, `XLM-RoBERTa`, `Longformer`, `BertForTokenClassification`, `DistilBertForTokenClassification`, `AlbertForTokenClassification`, `RoBertaForTokenClassification`, `DeBertaForTokenClassification`, `XlmRoBertaForTokenClassification`, `XlnetForTokenClassification`, `LongformerForTokenClassification`, `CamemBertForTokenClassification`, `BertForSequenceClassification`, `DistilBertForSequenceClassification`, `AlbertForSequenceClassification`, `RoBertaForSequenceClassification`, `DeBertaForSequenceClassification`, `XlmRoBertaForSequenceClassification`, `XlnetForSequenceClassification`, `LongformerForSequenceClassification`, `AlbertForQuestionAnswering`, `BertForQuestionAnswering`, `DeBertaForQuestionAnswering`, `DistilBertForQuestionAnswering`, `LongformerForQuestionAnswering`, `RoBertaForQuestionAnswering`, `XlmRoBertaForQuestionAnswering`, `TapasForQuestionAnswering`, and `Vision Transformers (ViT)` models to Spark NLP. We will work on the remaining annotators and extend this support to the rest with each release 😊 +We have extended support for `HuggingFace` 🤗 and `TF Hub` exported models since `3.1.0` to equivalent Spark NLP 🚀 annotators. Starting this release, you can easily use the `saved_model` feature in HuggingFace within a few lines of codes and import any `BERT`, `DistilBERT`, `CamemBERT`, `RoBERTa`, `DeBERTa`, `XLM-RoBERTa`, `Longformer`, `BertForTokenClassification`, `DistilBertForTokenClassification`, `AlbertForTokenClassification`, `RoBertaForTokenClassification`, `DeBertaForTokenClassification`, `XlmRoBertaForTokenClassification`, `XlnetForTokenClassification`, `LongformerForTokenClassification`, `CamemBertForTokenClassification`, `CamemBertForSequenceClassification`, `BertForSequenceClassification`, `DistilBertForSequenceClassification`, `AlbertForSequenceClassification`, `RoBertaForSequenceClassification`, `DeBertaForSequenceClassification`, `XlmRoBertaForSequenceClassification`, `XlnetForSequenceClassification`, `LongformerForSequenceClassification`, `AlbertForQuestionAnswering`, `BertForQuestionAnswering`, `DeBertaForQuestionAnswering`, `DistilBertForQuestionAnswering`, `LongformerForQuestionAnswering`, `RoBertaForQuestionAnswering`, `XlmRoBertaForQuestionAnswering`, `TapasForQuestionAnswering`, and `Vision Transformers (ViT)` models to Spark NLP. We will work on the remaining annotators and extend this support to the rest with each release 😊 ### Compatibility @@ -65,6 +65,7 @@ XlmRoBertaForTokenClassification | | ✅ | [TFXLMRobertaForTokenClassificati XlnetForTokenClassification | | ✅ | [TFXLNetForTokenClassificationet](https://huggingface.co/docs/transformers/model_doc/xlnet#transformers.TFXLNetForTokenClassificationet) LongformerForTokenClassification | | ✅ | [TFLongformerForTokenClassification](https://huggingface.co/docs/transformers/model_doc/longformer#transformers.TFLongformerForTokenClassification) CamemBertForTokenClassification | | ✅ | [TFCamemBertForTokenClassification](https://huggingface.co/docs/transformers/model_doc/camembert#transformers.TFCamembertForTokenClassification) +CamemBertForSequenceClassification | | ✅ | [TFCamemBertForSequenceClassification](https://huggingface.co/docs/transformers/model_doc/camembert#transformers.TFCamembertForSequenceClassification) BertForSequenceClassification | | ✅ | [TFBertForSequenceClassification](https://huggingface.co/docs/transformers/model_doc/bert#transformers.TFBertForSequenceClassification) DistilBertForSequenceClassification | | ✅ | [TFDistilBertForSequenceClassification](https://huggingface.co/docs/transformers/model_doc/distilbert#transformers.TFDistilBertForSequenceClassification) AlbertForSequenceClassification | | ✅ | [TFAlbertForSequenceClassification](https://huggingface.co/docs/transformers/model_doc/albert#transformers.TFAlbertForSequenceClassification) @@ -81,11 +82,11 @@ LongformerForSequenceClassification | | ✅ | [TFLongformerForSequenceClassi | RoBertaForQuestionAnswering | |✅ | [TFRobertaForQuestionAnswering](https://huggingface.co/docs/transformers/model_doc/roberta#transformers.TFRobertaForQuestionAnswering) | XlmRoBertaForQuestionAnswering | |✅ | [TFXLMRobertaForQuestionAnswering](https://huggingface.co/docs/transformers/model_doc/xlm-roberta#transformers.TFXLMRobertaForQuestionAnswering) | TapasForQuestionAnswering | | ❎ | [TFTapasForQuestionAnswering](https://huggingface.co/docs/transformers/model_doc/tapas#transformers.TFTapasForQuestionAnswering) -ViTForImageClassification | ❌ | ✅ | [TFViTForImageClassification](https://huggingface.co/docs/transformers/model_doc/vit#transformers.TFViTForImageClassification) -Automatic Speech Recognition (Wav2Vec2ForCTC)| | ❎ | [TFWav2Vec2ForCTC](https://huggingface.co/docs/transformers/model_doc/wav2vec2#transformers.TFWav2Vec2ForCTC) -T5Transformer | | ❌ | -MarianTransformer| | ❌ | -OpenAI GPT2| | ❌ | +| ViTForImageClassification | ❌ | ✅ | [TFViTForImageClassification](https://huggingface.co/docs/transformers/model_doc/vit#transformers.TFViTForImageClassification) +| Automatic Speech Recognition (Wav2Vec2ForCTC)| | ❎ | [TFWav2Vec2ForCTC](https://huggingface.co/docs/transformers/model_doc/wav2vec2#transformers.TFWav2Vec2ForCTC) +| T5Transformer | | ❌ | +| MarianTransformer| | ❌ | +| OpenAI GPT2| | ❌ | ### Example Notebooks @@ -109,6 +110,7 @@ AlbertForTokenClassification|[HuggingFace in Spark NLP - AlbertForTokenClassific RoBertaForTokenClassification|[HuggingFace in Spark NLP - RoBertaForTokenClassification](https://github.com/JohnSnowLabs/spark-nlp-workshop/blob/master/jupyter/transformers/HuggingFace%20in%20Spark%20NLP%20-%20RoBertaForTokenClassification.ipynb) | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/JohnSnowLabs/spark-nlp-workshop/blob/master/jupyter/transformers/HuggingFace%20in%20Spark%20NLP%20-%20RoBertaForTokenClassification.ipynb) XlmRoBertaForTokenClassification|[HuggingFace in Spark NLP - XlmRoBertaForTokenClassification](https://github.com/JohnSnowLabs/spark-nlp-workshop/blob/master/jupyter/transformers/HuggingFace%20in%20Spark%20NLP%20-%20XlmRoBertaForTokenClassification.ipynb) | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/JohnSnowLabs/spark-nlp-workshop/blob/master/jupyter/transformers/HuggingFace%20in%20Spark%20NLP%20-%20XlmRoBertaForTokenClassification.ipynb) CamemBertForTokenClassification|[HuggingFace in Spark NLP - CamemBertForTokenClassification](https://github.com/JohnSnowLabs/spark-nlp-workshop/blob/master/jupyter/transformers/HuggingFace%20in%20Spark%20NLP%20-%20CamemBertForTokenClassification.ipynb) | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/JohnSnowLabs/spark-nlp-workshop/blob/master/jupyter/transformers/HuggingFace%20in%20Spark%20NLP%20-%20CamemBertForTokenClassification.ipynb) +CamemBertForTokenClassification|[HuggingFace in Spark NLP - CamemBertForSequenceClassification](https://github.com/JohnSnowLabs/spark-nlp-workshop/blob/master/jupyter/transformers/HuggingFace%20in%20Spark%20NLP%20-%20CamemBertForSequenceClassification.ipynb) | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/JohnSnowLabs/spark-nlp-workshop/blob/master/jupyter/transformers/HuggingFace%20in%20Spark%20NLP%20-%20CamemBertForSequenceClassification.ipynb) BertForSequenceClassification |[HuggingFace in Spark NLP - BertForSequenceClassification](https://github.com/JohnSnowLabs/spark-nlp-workshop/blob/master/jupyter/transformers/HuggingFace%20in%20Spark%20NLP%20-%20BertForSequenceClassification.ipynb) | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/JohnSnowLabs/spark-nlp-workshop/blob/master/jupyter/transformers/HuggingFace%20in%20Spark%20NLP%20-%20BertForSequenceClassification.ipynb) DistilBertForSequenceClassification |[HuggingFace in Spark NLP - DistilBertForSequenceClassification](https://github.com/JohnSnowLabs/spark-nlp-workshop/blob/master/jupyter/transformers/HuggingFace%20in%20Spark%20NLP%20-%20DistilBertForSequenceClassification.ipynb) | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/JohnSnowLabs/spark-nlp-workshop/blob/master/jupyter/transformers/HuggingFace%20in%20Spark%20NLP%20-%20DistilBertForSequenceClassification.ipynb) AlbertForSequenceClassification |[HuggingFace in Spark NLP - AlbertForSequenceClassification](https://github.com/JohnSnowLabs/spark-nlp-workshop/blob/master/jupyter/transformers/HuggingFace%20in%20Spark%20NLP%20-%20BertForSequenceClassification.ipynb) | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/JohnSnowLabs/spark-nlp-workshop/blob/master/jupyter/transformers/HuggingFace%20in%20Spark%20NLP%20-%20AlbertForSequenceClassification.ipynb) diff --git a/python/README.md b/python/README.md index 6c16f6915b7db7..60117a2d35ed0f 100644 --- a/python/README.md +++ b/python/README.md @@ -271,6 +271,8 @@ Spark NLP 4.2.5 has been tested and is compatible with the following runtimes: - 11.1 ML - 11.2 - 11.2 ML +- 11.3 +- 11.3 ML **GPU:** @@ -283,6 +285,7 @@ Spark NLP 4.2.5 has been tested and is compatible with the following runtimes: - 11.0 ML & GPU - 11.1 ML & GPU - 11.2 ML & GPU +- 11.3 ML & GPU NOTE: Spark NLP 4.0.x is based on TensorFlow 2.7.x which is compatible with CUDA11 and cuDNN 8.0.2. The only Databricks runtimes supporting CUDA 11 are 9.x and above as listed under GPU.