Merge pull request #35 from toshiakit/dev

Support the latest models released by OpenAI on April 9, 2024

contents/presets.csv

@@ -2,18 +2,9 @@ name,content,prompt,model,max_tokens,temperature,test_code,suggested_questions,s
 AI Assistant,You are a helpful assistant. Answer as concisely as possible. ,Where is the capital of France?,gpt-3.5-turbo,1000,1,0,1,1
 Read a web page,You are a helpful assistant that can read a small web page and analyze its content. The content of the page will be extracted by an external function and stored in the chat history. ,What is this page about? https://www.mathworks.com/help/matlab/text-files.html,gpt-3.5-turbo,1000,0,0,1,1
 Read a local file,You are a helpful assistant that can read a small file and analyze its content. The content of the file will be extracted by an external function and stored in the chat history. ,Select a file using the paper clip icon on the left.,gpt-3.5-turbo,1000,0,0,1,1
-Understand an image,You are a helpful assistant that can see an image and analyze its content.,What is in this image? https://www.mathworks.com/help/examples/matlab/win64/DisplayGrayscaleRGBIndexedOrBinaryImageExample_04.png,gpt-4-vision-preview,1000,1,0,0,0
-Generate an image,,Create a 3D avatar of a whimsical sushi on the beach. He is decorated with various sushi elements and is playfully interacting with the beach environment.,dall-e-3,Inf,0,0,0,0
+Understand an image,You are a helpful assistant that can see an image and analyze its content.,What is in this image? https://www.mathworks.com/help/examples/matlab/win64/DisplayGrayscaleRGBIndexedOrBinaryImageExample_04.png,gpt-4-turbo,1000,1,0,1,1
+Generate an image,You are an AI assistant that can generate an image from a text prompt.,"An adventurous man clad in athletic gear is kiteboarding, harnessing the wind with his kite to skim across the waters of the famed Charles River in Boston. Nearby, a picturesque sailboat dances with the breeze. The city's skyline paints a stunning backdrop to the scene, while the surface of the river mirrors the golden hues of the setting sun. ",dall-e-3,4096,1,0,0,0
 English to MATLAB Code,You are a helpful assistant that generates MATLAB code. ,"Define two random vectors x and y, fit a linear model to the data, and plot both the data and fitted line.",gpt-3.5-turbo,1000,0,1,1,1
-English to Simulink Model,"You are a helpful assistant that creates Simulink models.  
-You create the models by generating MATLAB code that adds all the necessary blocks and set their parameters. 
-Automatically arrange the blocks by adding this line of code:
-Simulink.BlockDiagram.arrangeSystem(model)
-Save the model and run it.","Model name: 'sine_multiplied' 
-Add Sine Wave block with amplification = 1. 
-Multiply the sine wave signal by 3. 
-Add Scope block with 2 input ports. 
-Visualize both signals by connecting them to the same Scope block. ",gpt-3.5-turbo,1000,0,1,1,1
 Summarize Code,You are a friendly and helpful teaching assistant for MATLAB programmers. Analyze MATLAB code and provide concise summary of what the code does. ,"Summarize what the following code does. Code by Cecelya Blooming Light in MATLAB Mini Hack 2022
 
 ```matlab

helpers/MsgHelper.m

@@ -9,15 +9,14 @@
             % provide context window for a given model
             % supported models
             models = [ ...
-                struct('name','gpt-3.5-turbo','attributes',struct('contextwindow',4096,'cutoff','Sep 2021'),'legacy',false), ...
-                struct('name','gpt-3.5-turbo-1106','attributes',struct('contextwindow',16385,'cutoff','Sep 2021'),'legacy',false), ...
-                struct('name','gpt-3.5-turbo-16k','attributes',struct('contextwindow',16385,'cutoff','Sep 2021'),'legacy',false), ...
-                struct('name','gpt-4','attributes',struct('contextwindow',8192,'cutoff','Sep 2021'),'legacy',false), ...
-                struct('name','gpt-4-0613','attributes',struct('contextwindow',8192,'cutoff','Sep 2021'),'legacy',false), ...
-                struct('name','gpt-4-1106-preview','attributes',struct('contextwindow',128000,'cutoff','Apr 2023'),'legacy',false), ...
-                struct('name','gpt-4-vision-preview','attributes',struct('contextwindow',128000,'cutoff','Apr 2023'),'legacy',false), ...
-                struct('name','gpt-4-turbo-preview','attributes',struct('contextwindow',128000,'cutoff','Apr 2023'),'legacy',false), ...
-                struct('name','dall-e-3','attributes',struct('contextwindow','n/a','cutoff','n/a'),'legacy',false), ...
+                struct('name','gpt-3.5-turbo','attributes',struct('contextwindow',16385,'cutoff','Sep 2021')), ...
+                struct('name','gpt-3.5-turbo-0125','attributes',struct('contextwindow',16385,'cutoff','Sep 2021')), ...
+                struct('name','gpt-3.5-turbo-1106','attributes',struct('contextwindow',16385,'cutoff','Sep 2021')), ...
+                struct('name','gpt-4','attributes',struct('contextwindow',8192,'cutoff','Sep 2021')), ...
+                struct('name','gpt-4-0613','attributes',struct('contextwindow',8192,'cutoff','Sep 2021')), ...
+                struct('name','gpt-4-turbo-2024-04-09','attributes',struct('contextwindow',128000,'cutoff','Dec 2023')), ...
+                struct('name','gpt-4-turbo','attributes',struct('contextwindow',128000,'cutoff','Dec 2023')), ...
+                struct('name','dall-e-3','attributes',struct('contextwindow',16385,'cutoff','Sep 2021')), ...
                 ];
             contextwindow = models(arrayfun(@(x) string(x.name), models) == modelName).attributes.contextwindow;
             cutoff = models(arrayfun(@(x) string(x.name), models) == modelName).attributes.cutoff;
@@ -77,7 +76,8 @@
 
             % extract content from messages
             contents = strings(size(messages.Messages));
-            isText = ~cellfun(@(x) isempty(x.content), messages.Messages);
+            isText = cellfun(@(x) isstring(x.content), messages.Messages);
+            isText(cellfun(@(x) isempty(x.content), messages.Messages(isText))) = 0;
             contents(isText) = cellfun(@(x) x.content, messages.Messages(isText));
             isTestReport = startsWith(contents,'<div class="test-report">');
             if any(isTestReport)

helpers/imageGenGenerate.m

@@ -0,0 +1,29 @@
+function [image,message,response] = imageGenGenerate(chat,messages)
+% IMAGEGENGENERATE generates image using chat api with function calling
+
+    % generate a response to the prompt for image generation
+    [txt,message,response] = generate(chat,messages);
+    % if tool_call is returned
+    if isfield(message,"tool_calls")
+        toolCalls = message.tool_calls;
+        fcn = toolCalls.function.name;
+        % make sure it calls for 'generateImage'
+        if strcmp(fcn,"generateImage")
+            args = jsondecode(toolCalls.function.arguments);
+            prompt = string(args.prompt);
+            % execute 'generateImage' using the provided prompt
+            [image,revisedPrompt,response] = generateImage(prompt);
+            message = struct("role","assistant","content", string(revisedPrompt));
+        end
+    else
+        image = txt;
+    end
+end
+
+% helper function 'generateImage' to call DALL-E 3
+function [image, revisedPrompt ,response] = generateImage(prompt)
+    mdl = openAIImages(ModelName="dall-e-3");
+    [images, response] = generate(mdl,string(prompt));
+    image = images{1};
+    revisedPrompt = response.Body.Data.data.revised_prompt;
+end

helpers/llms-with-matlab/+llms/+internal/callOpenAIChatAPI.m

@@ -119,18 +119,16 @@
 
 parameters.stream = ~isempty(nvp.StreamFun);
 
-if ~isempty(functions) && ~strcmp(nvp.ModelName,'gpt-4-vision-preview')
+if ~isempty(functions)
     parameters.tools = functions;
 end
 
-if ~isempty(nvp.ToolChoice) && ~strcmp(nvp.ModelName,'gpt-4-vision-preview')
+if ~isempty(nvp.ToolChoice)
     parameters.tool_choice = nvp.ToolChoice;
 end
 
-if ismember(nvp.ModelName,["gpt-3.5-turbo-1106","gpt-4-1106-preview"])
-    if strcmp(nvp.ResponseFormat,"json")
-        parameters.response_format = struct('type','json_object');
-    end
+if strcmp(nvp.ResponseFormat,"json")
+    parameters.response_format = struct('type','json_object');
 end
 
 if ~isempty(nvp.Seed)
@@ -142,15 +140,21 @@
 dict = mapNVPToParameters;
 
 nvpOptions = keys(dict);
-if strcmp(nvp.ModelName,'gpt-4-vision-preview')
-    nvpOptions(ismember(nvpOptions,"StopSequences")) = [];
-end
 
 for opt = nvpOptions.'
     if isfield(nvp, opt)
         parameters.(dict(opt)) = nvp.(opt);
     end
 end
+
+if isempty(nvp.StopSequences)
+    parameters = rmfield(parameters,"stop");
+end
+
+if nvp.MaxNumTokens == Inf
+    parameters = rmfield(parameters,"max_tokens");
+end
+
 end
 
 function dict = mapNVPToParameters()

helpers/llms-with-matlab/+llms/+utils/errorMessageCatalog.m

@@ -49,8 +49,10 @@
 catalog("llms:mustBeMessagesOrTxt") = "Messages must be text with one or more characters or an openAIMessages objects.";
 catalog("llms:invalidOptionAndValueForModel") = "'{1}' with value '{2}' is not supported for ModelName '{3}'";
 catalog("llms:invalidOptionForModel") = "{1} is not supported for ModelName '{2}'";
+catalog("llms:invalidContentTypeForModel") = "{1} is not supported for ModelName '{2}'";
 catalog("llms:functionNotAvailableForModel") = "This function is not supported for ModelName '{1}'";
 catalog("llms:promptLimitCharacter") = "Prompt must have a maximum length of {1} characters for ModelName '{2}'";
 catalog("llms:pngExpected") = "Argument must be a PNG image.";
 catalog("llms:warningJsonInstruction") = "When using JSON mode, you must also prompt the model to produce JSON yourself via a system or user message.";
+catalog("llms:apiReturnedError") = "OpenAI API Error: {1}";
 end

helpers/llms-with-matlab/README.md

@@ -5,20 +5,36 @@ This repository contains example code to demonstrate how to connect MATLAB to th
 The functionality shown here serves as an interface to the ChatGPT and DALL·E APIs. To start using the OpenAI APIs, you first need to obtain OpenAI API keys. You are responsible for any fees OpenAI may charge for the use of their APIs. You should be familiar with the limitations and risks associated with using this technology, and you agree that you shall be solely responsible for full compliance with any terms that may apply to your use of the OpenAI APIs.
 
 Some of the current LLMs supported are:
-- gpt-3.5-turbo, gpt-3.5-turbo-1106
-- gpt-4, gpt-4-1106-preview
-- gpt-4-vision-preview (a.k.a. GPT-4 Turbo with Vision)
+- gpt-3.5-turbo, gpt-3.5-turbo-1106, gpt-3.5-turbo-0125
+- gpt-4-turbo, gpt-4-turbo-2024-04-09 (GPT-4 Turbo with Vision)
+- gpt-4, gpt-4-0613
 - dall-e-2, dall-e-3
-
+                                                        
 For details on the specification of each model, check the official [OpenAI documentation](https://platform.openai.com/docs/models).
 
+## Requirements
+
+### MathWorks Products (https://www.mathworks.com)
+
+- Requires MATLAB release R2024a or newer.
+- Some examples require Text Analytics Toolbox™.
+
+### 3rd Party Products:
+
+- An active OpenAI API subscription and API key.
+
 ## Setup
 
-If you would like to use this repository with MATLAB Online, simply click [![Open in MATLAB Online](https://www.mathworks.com/images/responsive/global/open-in-matlab-online.svg)](https://matlab.mathworks.com/open/github/v1?repo=matlab-deep-learning/llms-with-matlab)
+### MATLAB Online
+
+To use this repository with MATLAB Online, click [![Open in MATLAB Online](https://www.mathworks.com/images/responsive/global/open-in-matlab-online.svg)](https://matlab.mathworks.com/open/github/v1?repo=matlab-deep-learning/llms-with-matlab)
+
+
+### MATLAB Desktop
 
-If you would like to use it with MATLAB Desktop, proceed with the following steps:
+To use this repository with a local installation of MATLAB, first clone the repository. 
 
-1. Clone the repository to your local machine.
+1. In the system command prompt, run:
 
     ```bash
     git clone https://github.com/matlab-deep-learning/llms-with-matlab.git
@@ -32,26 +48,19 @@ If you would like to use it with MATLAB Desktop, proceed with the following step
     addpath('path/to/llms-with-matlab');
     ```
 
-4. Set up your OpenAI API key. Create a `.env` file in the project root directory with the following content.
+### Setting up your API key
 
-    ```
-    OPENAI_API_KEY=<your key>
-    ```
-
-    Then load your `.env` file as follows:
+Set up your OpenAI API key. Create a `.env` file in the project root directory with the following content.
 
-    ```matlab
-    loadenv(".env")
-    ```
-
-### MathWorks Products (https://www.mathworks.com)
-
-- Requires MATLAB release R2023a or newer.
-
-### 3rd Party Products:
-
-- An active OpenAI API subscription and API key.
+```
+OPENAI_API_KEY=<your key>
+```
+    
+Then load your `.env` file as follows:
 
+```matlab
+loadenv(".env")
+```
 
 ## Getting Started with Chat Completion API
 
@@ -278,13 +287,13 @@ You can extract the arguments and write the data to a table, for example.
 
 ### Understand the content of an image
 
-You can use gpt-4-vision-preview to experiment with image understanding. 
+You can use gpt-4-turbo to experiment with image understanding. 
 ```matlab
-chat = openAIChat("You are an AI assistant.", ModelName="gpt-4-vision-preview"); 
+chat = openAIChat("You are an AI assistant.", ModelName="gpt-4-turbo");
 image_path = "peppers.png";
 messages = openAIMessages;
 messages = addUserMessageWithImages(messages,"What is in the image?",image_path);
-[txt,response] = generate(chat,messages);
+[txt,response] = generate(chat,messages,MaxNumTokens=4096);
 % Should output the description of the image
 ```
 
@@ -320,15 +329,16 @@ imshow(images{1})
 ## Examples
 To learn how to use this in your workflows, see [Examples](/examples/).
 
-- [ExampleStreaming.mlx](/examples/ExampleStreaming.mlx): Learn to implement a simple chat that stream the response. 
-- [ExampleSummarization.mlx](/examples/ExampleSummarization.mlx): Learn to create concise summaries of long texts with ChatGPT. (Requires Text Analytics Toolbox™)
-- [ExampleChatBot.mlx](/examples/ExampleChatBot.mlx): Build a conversational chatbot capable of handling various dialogue scenarios using ChatGPT. (Requires Text Analytics Toolbox)
-- [ExampleFunctionCalling.mlx](/examples/ExampleFunctionCalling.mlx): Learn how to create agents capable of executing MATLAB functions. 
+- [ProcessGeneratedTextinRealTimebyUsingChatGPTinStreamingMode.mlx](/examples/ProcessGeneratedTextinRealTimebyUsingChatGPTinStreamingMode.mlx): Learn to implement a simple chat that stream the response. 
+- [SummarizeLargeDocumentsUsingChatGPTandMATLAB.mlx](/examples/SummarizeLargeDocumentsUsingChatGPTandMATLAB.mlx): Learn to create concise summaries of long texts with ChatGPT. (Requires Text Analytics Toolbox™)
+- [CreateSimpleChatBot.mlx](/examples/CreateSimpleChatBot.mlx): Build a conversational chatbot capable of handling various dialogue scenarios using ChatGPT. (Requires Text Analytics Toolbox)
+- [AnalyzeScientificPapersUsingFunctionCalls.mlx](/examples/AnalyzeScientificPapersUsingFunctionCalls.mlx): Learn how to create agents capable of executing MATLAB functions. 
 - [ExampleParallelFunctionCalls.mlx](/examples/ExampleParallelFunctionCalls.mlx): Learn how to take advantage of parallel function calling. 
-- [ExampleRetrievalAugmentedGeneration.mlx](/examples/ExampleRetrievalAugmentedGeneration.mlx): Learn about retrieval augmented generation with a simple use case. (Requires Text Analytics Toolbox™)
-- [ExampleGPT4Vision.mlx](/examples/ExampleGPT4Vision.mlx): Learn how to use GPT-4 Turbo with Vision to understand the content of an image. 
-- [ExampleJSONMode.mlx](/examples/ExampleJSONMode.mlx): Learn how to use JSON mode in chat completions
-- [ExampleDALLE.mlx](/examples/ExampleDALLE.mlx): Learn how to generate images, create variations and edit the images. 
+- [RetrievalAugmentedGenerationUsingChatGPTandMATLAB.mlx](/examples/RetrievalAugmentedGenerationUsingChatGPTandMATLAB.mlx): Learn about retrieval augmented generation with a simple use case. (Requires Text Analytics Toolbox™)
+- [DescribeImagesUsingChatGPT.mlx](/examples/DescribeImagesUsingChatGPT.mlx): Learn how to use GPT-4 Turbo with Vision to understand the content of an image. 
+- [AnalyzeSentimentinTextUsingChatGPTinJSONMode.mlx](/examples/AnalyzeSentimentinTextUsingChatGPTinJSONMode.mlx): Learn how to use JSON mode in chat completions
+- [UsingDALLEToEditImages.mlx](/examples/UsingDALLEToEditImages.mlx): Learn how to generate images
+- [UsingDALLEToGenerateImages.mlx](/examples/UsingDALLEToGenerateImages.mlx): Create variations of images and editimages. 
 
 ## License