Merge pull request #36 from toshiakit/dev-2024-05-08

Incorporates the latest update on LLMs with MATLAB and supports Voice Chat

README.md

@@ -19,12 +19,13 @@ MatGPT was updated to run on the framework from the '[Large Language Models (LLM
 * MatGPT detects a URL included in a prompt, and retrieves its web content into the chat.
 * MatGPT lets you import a .m, .mlx, .csv or .txt file into the chat. PDF files are also supported if Text Analytics Toolbox is available.
 * MatGPT supports GPT-4 Turbo with Vision. You can pass the URL to an image, or a local image file path ask questions about the image.
-* MatGPT lets you generate an image via DALL·E 3. 
+* MatGPT lets you generate an image via DALL·E 3 API. 
+* MatGPT supports voice chat via Whisper API. 
 
 Please note that:
 
 * imported content will be truncated if it exceeds the context window limit.
-* Streaming must be disabled to use image generation via DALL·E 3. 
+* Streaming must be disabled to use image generation via DALL·E 3.
 
 ## Requirements
 

helpers/MsgHelper.m

@@ -16,6 +16,8 @@
                 struct('name','gpt-4-0613','attributes',struct('contextwindow',8192,'cutoff','Sep 2021')), ...
                 struct('name','gpt-4-turbo-2024-04-09','attributes',struct('contextwindow',128000,'cutoff','Dec 2023')), ...
                 struct('name','gpt-4-turbo','attributes',struct('contextwindow',128000,'cutoff','Dec 2023')), ...
+                struct('name','gpt-4o','attributes',struct('contextwindow',128000,'cutoff','Oct 2023')), ...
+                struct('name','gpt-4o-2024-05-13','attributes',struct('contextwindow',128000,'cutoff','Oct 2023')), ...
                 struct('name','dall-e-3','attributes',struct('contextwindow',16385,'cutoff','Sep 2021')), ...
                 ];
             contextwindow = models(arrayfun(@(x) string(x.name), models) == modelName).attributes.contextwindow;
@@ -38,7 +40,8 @@
                 "<li>Detects a URL included in a prompt, and retrieve its content into the chat.</<li>" + ...
                 "<li>Lets you import a .m, .mlx, .csv, or .txt file into the chat</li>" + ...
                 "<li>Supports GPT-4 Turbo with Vision with .jpg, .png or .gif images</li>" + ...
-                "<li>Supports image generation via DALLe-3 API when streaming is disabled</li></ul>" + ...
+                "<li>Supports image generation via DALLe-3 API when streaming is disabled</li>" + ...
+                "<li>Supports voice chat via Whisper API</li></ul>" + ...
                 "<strong>Notes</strong>:" + ...
                 "<ul><li>Imported content will be truncated if it exceeds the context window limit.</li>" + ...
                 "<li>PDF file is supported if Text Analytics Toolbox is available.</li></ul>"];

helpers/llms-with-matlab/+llms/+utils/errorMessageCatalog.m

@@ -55,4 +55,5 @@
 catalog("llms:pngExpected") = "Argument must be a PNG image.";
 catalog("llms:warningJsonInstruction") = "When using JSON mode, you must also prompt the model to produce JSON yourself via a system or user message.";
 catalog("llms:apiReturnedError") = "OpenAI API Error: {1}";
+catalog("llms:dimensionsMustBeSmallerThan") = "Dimensions must be less than or equal to {1}.";
 end

helpers/llms-with-matlab/README.md

@@ -1,4 +1,6 @@
-# Large Language Models (LLMs) with MATLAB® [![Open in MATLAB Online](https://www.mathworks.com/images/responsive/global/open-in-matlab-online.svg)](https://matlab.mathworks.com/open/github/v1?repo=matlab-deep-learning/llms-with-matlab)
+# Large Language Models (LLMs) with MATLAB®
+
+[![Open in MATLAB Online](https://www.mathworks.com/images/responsive/global/open-in-matlab-online.svg)](https://matlab.mathworks.com/open/github/v1?repo=matlab-deep-learning/llms-with-matlab) [![View Large Language Models (LLMs) with MATLAB on File Exchange](https://www.mathworks.com/matlabcentral/images/matlab-file-exchange.svg)](https://www.mathworks.com/matlabcentral/fileexchange/163796-large-language-models-llms-with-matlab) 
 
 This repository contains example code to demonstrate how to connect MATLAB to the OpenAI™ Chat Completions API (which powers ChatGPT™) as well as OpenAI Images API (which powers DALL·E™). This allows you to leverage the natural language processing capabilities of large language models directly within your MATLAB environment.
 
@@ -142,7 +144,7 @@ txt = generate(chat,"What is Model-Based Design and how is it related to Digital
 ### Calling MATLAB functions with the API
 
 Optionally, `Tools=functions` can be used to provide function specifications to the API. The purpose of this is to enable models to generate function arguments which adhere to the provided specifications. 
-Note that the API is not able to directly call any function, so you should call the function and pass the values to the API directly. This process can be automated as shown in [ExampleFunctionCalling.mlx](/examples/ExampleFunctionCalling.mlx), but it's important to consider that ChatGPT can hallucinate function names, so avoid executing any arbitrary generated functions and only allow the execution of functions that you have defined. 
+Note that the API is not able to directly call any function, so you should call the function and pass the values to the API directly. This process can be automated as shown in [AnalyzeScientificPapersUsingFunctionCalls.mlx](/examples/AnalyzeScientificPapersUsingFunctionCalls.mlx), but it's important to consider that ChatGPT can hallucinate function names, so avoid executing any arbitrary generated functions and only allow the execution of functions that you have defined. 
 
 For example, if you want to use the API for mathematical operations such as `sind`, instead of letting the model generate the result and risk running into hallucinations, you can give the model direct access to the function as follows:
 
@@ -333,7 +335,7 @@ To learn how to use this in your workflows, see [Examples](/examples/).
 - [SummarizeLargeDocumentsUsingChatGPTandMATLAB.mlx](/examples/SummarizeLargeDocumentsUsingChatGPTandMATLAB.mlx): Learn to create concise summaries of long texts with ChatGPT. (Requires Text Analytics Toolbox™)
 - [CreateSimpleChatBot.mlx](/examples/CreateSimpleChatBot.mlx): Build a conversational chatbot capable of handling various dialogue scenarios using ChatGPT. (Requires Text Analytics Toolbox)
 - [AnalyzeScientificPapersUsingFunctionCalls.mlx](/examples/AnalyzeScientificPapersUsingFunctionCalls.mlx): Learn how to create agents capable of executing MATLAB functions. 
-- [ExampleParallelFunctionCalls.mlx](/examples/ExampleParallelFunctionCalls.mlx): Learn how to take advantage of parallel function calling. 
+- [AnalyzeTextDataUsingParallelFunctionCallwithChatGPT.mlx](/examples/AnalyzeTextDataUsingParallelFunctionCallwithChatGPT.mlx): Learn how to take advantage of parallel function calling. 
 - [RetrievalAugmentedGenerationUsingChatGPTandMATLAB.mlx](/examples/RetrievalAugmentedGenerationUsingChatGPTandMATLAB.mlx): Learn about retrieval augmented generation with a simple use case. (Requires Text Analytics Toolbox™)
 - [DescribeImagesUsingChatGPT.mlx](/examples/DescribeImagesUsingChatGPT.mlx): Learn how to use GPT-4 Turbo with Vision to understand the content of an image. 
 - [AnalyzeSentimentinTextUsingChatGPTinJSONMode.mlx](/examples/AnalyzeSentimentinTextUsingChatGPTinJSONMode.mlx): Learn how to use JSON mode in chat completions

helpers/llms-with-matlab/extractOpenAIEmbeddings.m

@@ -20,14 +20,14 @@
 %   [emb, response] = EXTRACTOPENAIEMBEDDINGS(...) also returns the full
 %   response from the OpenAI API call.
 %
-%   Copyright 2023 The MathWorks, Inc.
+%   Copyright 2023-2024 The MathWorks, Inc.
 
 arguments
-    text           (1,:) {mustBeText}
+    text           (1,:) {mustBeNonzeroLengthText}
     nvp.ModelName  (1,1) {mustBeMember(nvp.ModelName,["text-embedding-ada-002", ...
                         "text-embedding-3-large", "text-embedding-3-small"])} = "text-embedding-ada-002"
     nvp.TimeOut    (1,1) {mustBeReal,mustBePositive} = 10
-    nvp.Dimensions (1,1) {mustBeInteger}
+    nvp.Dimensions (1,1) {mustBeInteger,mustBePositive}
     nvp.ApiKey           {llms.utils.mustBeNonzeroLengthTextScalar}
 end
 
@@ -42,6 +42,7 @@
         error("llms:invalidOptionForModel", ...
             llms.utils.errorMessageCatalog.getMessage("llms:invalidOptionForModel", "Dimensions", nvp.ModelName));
     end
+    mustBeCorrectDimensions(nvp.Dimensions,nvp.ModelName);
     parameters.dimensions = nvp.Dimensions;
 end
 
@@ -53,4 +54,17 @@
     emb = emb';
 else
     emb = [];
+end
+end
+
+function mustBeCorrectDimensions(dimensions,modelName)
+    model2dim = ....
+    dictionary(["text-embedding-3-large", "text-embedding-3-small"], ...
+    [3072,1536]);
+
+    if dimensions>model2dim(modelName)
+        error("llms:dimensionsMustBeSmallerThan", ...
+            llms.utils.errorMessageCatalog.getMessage("llms:dimensionsMustBeSmallerThan", ...
+            string(model2dim(modelName))));
+    end
 end

helpers/llms-with-matlab/openAIChat.m

@@ -114,7 +114,8 @@
             arguments
                 systemPrompt                       {llms.utils.mustBeTextOrEmpty} = []
                 nvp.Tools                    (1,:) {mustBeA(nvp.Tools, "openAIFunction")} = openAIFunction.empty
-                nvp.ModelName                (1,1) {mustBeMember(nvp.ModelName,["gpt-4-turbo", ...
+                nvp.ModelName                (1,1) {mustBeMember(nvp.ModelName,["gpt-4o", ...
+                                                        "gpt-4o-2024-05-13","gpt-4-turbo", ...
                                                         "gpt-4-turbo-2024-04-09","gpt-4","gpt-4-0613", ...
                                                         "gpt-3.5-turbo","gpt-3.5-turbo-0125", ...
                                                         "gpt-3.5-turbo-1106"])} = "gpt-3.5-turbo"
@@ -217,7 +218,7 @@
             end
 
             if iscell(messagesStruct{end}.content) && any(cellfun(@(x) isfield(x,"image_url"), messagesStruct{end}.content))
-                if ~ismember(this.ModelName,["gpt-4-turbo","gpt-4-turbo-2024-04-09"]) 
+                if ~ismember(this.ModelName,["gpt-4-turbo","gpt-4-turbo-2024-04-09","gpt-4o","gpt-4o-2024-05-13"]) 
                  error("llms:invalidContentTypeForModel", ...
                        llms.utils.errorMessageCatalog.getMessage("llms:invalidContentTypeForModel", "Image content", this.ModelName));
                 end

helpers/llms-with-matlab/openAIImages.m

@@ -85,8 +85,8 @@
             %                          Only "dall-e-3" supports this parameter.
 
             arguments
-                this                    (1,1)  openAIImages
-                prompt                        {mustBeTextScalar}
+                this                    (1,1) openAIImages
+                prompt                        {mustBeNonzeroLengthTextScalar}
                 nvp.NumImages           (1,1) {mustBePositive, mustBeInteger,...
                                                 mustBeLessThanOrEqual(nvp.NumImages,10)} = 1
                 nvp.Size                (1,1) string {mustBeMember(nvp.Size, ["256x256", "512x512", ...
@@ -176,7 +176,7 @@
             arguments
                 this                    (1,1)  openAIImages
                 imagePath                     {mustBeValidFileType(imagePath)}
-                prompt                        {mustBeTextScalar}
+                prompt                        {mustBeNonzeroLengthTextScalar}
                 nvp.MaskImagePath             {mustBeValidFileType(nvp.MaskImagePath)}
                 nvp.NumImages           (1,1) {mustBePositive, mustBeInteger,...
                                                 mustBeLessThanOrEqual(nvp.NumImages,10)} = 1
@@ -345,7 +345,9 @@ function validatePromptSize(model, prompt)
 function mustBeValidFileType(filePath)
     mustBeFile(filePath);
     s = dir(filePath);
-    if ~endsWith(s.name, ".png")
+    imgDetails = imfinfo(filePath);
+    imgFormat = imgDetails.Format;
+    if ~(imgFormat=="png")
         error("llms:pngExpected", ...
             llms.utils.errorMessageCatalog.getMessage("llms:pngExpected"));
     end