Merge pull request #29 from toshiakit/dev

Adding support for GPT-4 Vision & DALL·E Image Generation

.gitignore

@@ -2,4 +2,5 @@
 *.asv
 contents/ChatData.mat
 contents/GeneratedCode/
+contents/GeneratedImages/
 startup.m

README.md

@@ -16,11 +16,15 @@ MatGPT was updated to run on the framework from the '[Large Language Models (LLM
 
 * MatGPT runs on the 'LLMs with MATLAB' framework, which requires MATLAB R2023a or later. 
 * MatGPT supports streaming API where response tokens are displayed as they come in.
-* MatGPT detects a URL included in a prompt, and retrieve its web content into the chat.
+* MatGPT detects a URL included in a prompt, and retrieves its web content into the chat.
 * MatGPT lets you import a .m, .mlx, .csv or .txt file into the chat. PDF files are also supported if Text Analytics Toolbox is available.
-* MatGPT supports GPT-4 Turbo with Vision. You can pass the URL to an image or a local image file path ask questions about the image. 
+* MatGPT supports GPT-4 Turbo with Vision. You can pass the URL to an image, or a local image file path ask questions about the image.
+* MatGPT lets you generate an image via DALL·E 3. 
 
-Please note that imported content will be truncated if it exceeds the context window limit. 
+Please note that:
+
+* imported content will be truncated if it exceeds the context window limit.
+* Streaming must be disabled to use image generation via DALL·E 3. 
 
 ## Requirements
 
@@ -54,7 +58,7 @@ To use MatGPT on MATLAB Online, simply click [![Open in MATLAB Online](https://w
 * If you want suggestion for follow-up questions in the response, check `Suggest follow-up questions` checkbox. Suggested questions appear as clickable buttons. You can copy a suggested question to the prompt box by clicking it.  
 * If your prompt is intended to generate MATLAB code, check `Test Generated MATLAB Code` checkbox to test the returned code.
 * The `Usage` tab shows the number of tokens used in the current chat session. 
-* Add stop sequences in `Advanced` tab to specify the sequences where the API will stop generating further tokens
+* Add stop sequences in `Advanced` tab to specify the sequences where the API will stop generating further tokens.
 4. Continue the conversation by keep adding more prompts and clicking `Send`. 
 5. You can right-click or double-click a chat in the left navigation panel to rename, delete, or save the chat to a text file. 
 6. When you close the app, the chat will be saved and will be reloaded into the left nav when you relaunch the app.

contents/presets.csv

@@ -1,9 +1,10 @@
-name,content,prompt,model,max_tokens,temperature,test_code
-AI Assistant,You are a helpful assistant. Answer as concisely as possible. ,Where is the capital of France?,gpt-3.5-turbo,1000,1,0
-Read a web page,You are a helpful assistant that can read a small web page and analyze its content. The content of the page will be extracted by an external function and stored in the chat history. ,What is this page about? https://www.mathworks.com/help/matlab/text-files.html,gpt-3.5-turbo,1000,0,0
-Read a local file,You are a helpful assistant that can read a small file and analyze its content. The content of the file will be extracted by an external function and stored in the chat history. ,Select a file using the paper clip icon on the left.,gpt-3.5-turbo,1000,0,0
-Understand an image,You are a helpful assistant that can see an image and analyze its content.,What is in this image? https://www.mathworks.com/help/examples/matlab/win64/DisplayGrayscaleRGBIndexedOrBinaryImageExample_04.png,gpt-4-vision-preview,Inf,1,0
-English to MATLAB Code,You are a helpful assistant that generates MATLAB code. ,"Define two random vectors x and y, fit a linear model to the data, and plot both the data and fitted line.",gpt-3.5-turbo,1000,0,1
+name,content,prompt,model,max_tokens,temperature,test_code,suggested_questions,streaming
+AI Assistant,You are a helpful assistant. Answer as concisely as possible. ,Where is the capital of France?,gpt-3.5-turbo,1000,1,0,1,1
+Read a web page,You are a helpful assistant that can read a small web page and analyze its content. The content of the page will be extracted by an external function and stored in the chat history. ,What is this page about? https://www.mathworks.com/help/matlab/text-files.html,gpt-3.5-turbo,1000,0,0,1,1
+Read a local file,You are a helpful assistant that can read a small file and analyze its content. The content of the file will be extracted by an external function and stored in the chat history. ,Select a file using the paper clip icon on the left.,gpt-3.5-turbo,1000,0,0,1,1
+Understand an image,You are a helpful assistant that can see an image and analyze its content.,What is in this image? https://www.mathworks.com/help/examples/matlab/win64/DisplayGrayscaleRGBIndexedOrBinaryImageExample_04.png,gpt-4-vision-preview,1000,1,0,0,0
+Generate an image,,Create a 3D avatar of a whimsical sushi on the beach. He is decorated with various sushi elements and is playfully interacting with the beach environment.,dall-e-3,Inf,0,0,0,0
+English to MATLAB Code,You are a helpful assistant that generates MATLAB code. ,"Define two random vectors x and y, fit a linear model to the data, and plot both the data and fitted line.",gpt-3.5-turbo,1000,0,1,1,1
 English to Simulink Model,"You are a helpful assistant that creates Simulink models.  
 You create the models by generating MATLAB code that adds all the necessary blocks and set their parameters. 
 Automatically arrange the blocks by adding this line of code:
@@ -12,7 +13,7 @@ Save the model and run it.","Model name: 'sine_multiplied'
 Add Sine Wave block with amplification = 1. 
 Multiply the sine wave signal by 3. 
 Add Scope block with 2 input ports. 
-Visualize both signals by connecting them to the same Scope block. ",gpt-3.5-turbo,1000,0,1
+Visualize both signals by connecting them to the same Scope block. ",gpt-3.5-turbo,1000,0,1,1,1
 Summarize Code,You are a friendly and helpful teaching assistant for MATLAB programmers. Analyze MATLAB code and provide concise summary of what the code does. ,"Summarize what the following code does. Code by Cecelya Blooming Light in MATLAB Mini Hack 2022
 
 ```matlab
@@ -32,7 +33,7 @@ axis( 'equal','off')
 view([0 33]) 
 colormap(flip(hot)) 
 shading('interp')
-```",gpt-3.5-turbo,1000,0,0
+```",gpt-3.5-turbo,1000,0,0,1,1
 Explain Code Step by Step,You are a friendly and helpful teaching assistant for MATLAB programmers. Analyze MATLAB code and explain the code step by step.,"Explain what the following code does. Code by Cecelya Blooming Light in MATLAB Mini Hack 2022 
 
 ```matlab
@@ -52,7 +53,7 @@ axis( 'equal','off')
 view([0 33]) 
 colormap(flip(hot)) 
 shading('interp')
-```",gpt-3.5-turbo,1000,0,0
+```",gpt-3.5-turbo,1000,0,0,1,1
 Script to function,You are a friendly and helpful teaching assistant for MATLAB programmers. Convert MATLAB scripts to a local function and call the function using sample inputs.  ,"Convert the following script to a function, using m and T as input variables and set default values to those variables. Code by Cecelya Blooming Light in MATLAB Mini Hack 2022
 
 ```matlab
@@ -72,7 +73,7 @@ axis( 'equal','off')
 view([0 33]) 
 colormap(flip(hot)) 
 shading('interp')
-```",gpt-3.5-turbo,1000,0,1
+```",gpt-3.5-turbo,1000,0,1,1,1
 Write doc to function,"You are a friendly and helpful teaching assistant for MATLAB programmers. Analyze a MATLAB function and add an elaborate, high quality docstring to the function.","Write useful docstring to this function derived from the code, using comment format %PLOTBLOOMINGLIGHTS does blah blah blah.  Code by Cecelya Blooming Light in MATLAB Mini Hack 2022
 
 ```matlab
@@ -96,7 +97,7 @@ view([0 33])
 colormap(flip(hot)) 
 shading('interp')
 end
-```",gpt-3.5-turbo,1000,0,0
+```",gpt-3.5-turbo,1000,0,0,1,1
 Fix Bug,You are a friendly and helpful teaching assistant for MATLAB programmers. Analyze buggy MATLAB code and provide error free code using latest features from R2021a or later. ,"Fix bug in the following MATLAB code
 
 ```matlab
@@ -108,14 +109,14 @@ y = ""a"" + b
 end
 ```
 
-% Fixed MATLAB code",gpt-3.5-turbo,1000,0,0
+% Fixed MATLAB code",gpt-3.5-turbo,1000,0,0,1,1
 Write Unit Tests,You are a friendly and helpful teaching assistant for MATLAB programmers. Analyze MATLAB functions and write appropriate unit tests.,"Write unit tests for this function, 
 
 ```matlab
 function y = add(a, b)
 y = a + b
 end
-```",gpt-3.5-turbo,1000,0,0
+```",gpt-3.5-turbo,1000,0,0,1,1
 Vectorize Code,You are a friendly and helpful teaching assistant for MATLAB programmers. Analyze MATLAB code and rewrite it to improve efficiency using vectorization rather than for loops.,"Vectorize the following code without using for loop. 
 
 ```matlab
@@ -124,12 +125,12 @@ for t = 0:.01:10
     i = i + 1;
     y(i) = sin(t);
 end
-```",gpt-3.5-turbo,1000,0,1
+```",gpt-3.5-turbo,1000,0,1,1,1
 Act as MATLAB Command Window,"You are a MATLAB interpreter. I will give you MATLAB code, you'll reply with what the Command Window should show. Do not provide any explanations. Do not respond with anything except the output of the code. ","```matlab
 x = (1:10)*2
 ```
 
-",gpt-3.5-turbo,1000,0,0
+",gpt-3.5-turbo,1000,0,0,1,1
 Sentiment Analysis,You are a helpful assistant that helps data scientists by analyzing sentiment expressed in a given text.,"Decide whether sentiment is positive, neutral, or negative for the following text
 
 1. ""I can't stand homework""
@@ -138,7 +139,7 @@ Sentiment Analysis,You are a helpful assistant that helps data scientists by ana
 4. ""My cat is adorable ❤️❤️""
 5. ""I hate chocolate""
 
-Text sentiment ratings:",gpt-3.5-turbo,1000,0,0
+Text sentiment ratings:",gpt-3.5-turbo,1000,0,0,1,1
 Contact Extraction,You are a helpful assistant that helps data scientists by analyzing unstructured data and extract personally identifiable information.,"Extract the name and mailing address from this text: 
 
 Dear Kelly,
@@ -150,7 +151,7 @@ Thank you for the book. Here's my address
 
 Best,
 
-Maya",gpt-3.5-turbo,1000,0,0
+Maya",gpt-3.5-turbo,1000,0,0,1,1
 Extract table from PDF,You are a helpful assistant that helps data scientists by analyzing unstructured data and recreate a table from the text extracted from a PDF file.,"format the following PDF data into a table. Start with a title, then the table and end with a footer citing the source.
 
 A(Percentage)verage annual export growth rates of creative goods, 2006−2020
@@ -218,20 +219,20 @@ A(Percentage)verage annual export growth rates of creative goods, 2006−2020
      -7.2
 
 
-     ",gpt-3.5-turbo,1000,0,0
-Top 10 SciFi movies,You are a helpful assistant that helps data scientists extract useful public data for analysis.,"Create a  table of top 10 science fiction movies,  their year of release, and box office gross. Name columns 'Title', 'Year', and 'Gross'.",gpt-3.5-turbo,1000,0,0
+     ",gpt-3.5-turbo,1000,0,0,1,1
+Top 10 SciFi movies,You are a helpful assistant that helps data scientists extract useful public data for analysis.,"Create a  table of top 10 science fiction movies,  their year of release, and box office gross. Name columns 'Title', 'Year', and 'Gross'.",gpt-3.5-turbo,1000,0,0,1,1
 San Francisco Weather,You are a helpful assistant that helps data scientists extract useful public data for analysis.,"Create a table of weather temperatures for San Francisco.
 Name columns 'Month', 'High', and 'Low'. 
-Use abbreviated month names. Use numbers only for temperature using Fahrenheit.",gpt-3.5-turbo,1000,0,0
+Use abbreviated month names. Use numbers only for temperature using Fahrenheit.",gpt-3.5-turbo,1000,0,0,1,1
 Boston Housing Prices,You are a helpful assistant that helps data scientists extract useful public data for analysis.,"Housing prices are correlated with size of the house. 
 Create a table of housing prices in Boston based on square footage as size 
 with columns named 'Sqft' and 'Price' with 10 rows. Randomize the values in 'Sqft', 
-while maintaining the correlation to 'Price' with some variation.",gpt-3.5-turbo,1000,0,0
+while maintaining the correlation to 'Price' with some variation.",gpt-3.5-turbo,1000,0,0,1,1
 Boston Landmarks,You are a helpful assistant that helps data scientists extract useful public data for analysis.,"Create a markdown table of famous landmarks in Boston 
      with latitude and longitude in decimal degrees, i.e. 42.361145, -71.057083
      Generate MATLAB Code that stores the data in variables landmarks, lat and lon. 
      Use geoscatter function to plot the data points. Annotate the data points with the names of the landmarks using text function.
-     Avoid using for loop.",gpt-3.5-turbo,1000,0,0
+     Avoid using for loop.",gpt-3.5-turbo,1000,0,0,1,1
 Remove duplicate names,You are a helpful assistant that helps data scientists extract useful public data for analysis.,"Here is a list of names.
 
 John Smith
@@ -248,7 +249,7 @@ Brown Jennifer
 Create a table with two columns from this list 
 that split the first name and last name. 
 If first name appears as last name, correct the order.  
-remove any duplicates. Show the table in markdown format.",gpt-3.5-turbo,1000,0,0
+remove any duplicates. Show the table in markdown format.",gpt-3.5-turbo,1000,0,0,1,1
 Email Domains to Company Names,You are a helpful assistant that helps data scientists extract useful public data for analysis.,"Identify legal company names based on the following domains
 
 1. @apple.com
@@ -264,7 +265,7 @@ Email Domains to Company Names,You are a helpful assistant that helps data scien
 11. @mcdonalds.com
 12. @chevron.com1
 13. @unitedparcelservice.com
-14. @dell.com",gpt-3.5-turbo,1000,0,0
+14. @dell.com",gpt-3.5-turbo,1000,0,0,1,1
 Normalize University Names,You are a helpful assistant that helps data scientists extract useful public data for analysis.,"Normalize the list of university names below. If a university has multiple names, apply the most common full name, not an abbreviation.
 
 CMU
@@ -288,4 +289,4 @@ Georgiatech
 MIT
 Massachusetts Institute of Technology
 
-",gpt-3.5-turbo,1000,0,0
+",gpt-3.5-turbo,1000,0,0,1,1

helpers/MsgHelper.m

@@ -16,7 +16,8 @@
                 struct('name','gpt-4-0613','attributes',struct('contextwindow',8192,'cutoff','Sep 2021'),'legacy',false), ...
                 struct('name','gpt-4-1106-preview','attributes',struct('contextwindow',128000,'cutoff','Apr 2023'),'legacy',false), ...
                 struct('name','gpt-4-vision-preview','attributes',struct('contextwindow',128000,'cutoff','Apr 2023'),'legacy',false), ...
-                struct('name','gpt-4-turbo-preview','attributes',struct('contextwindow',128000,'cutoff','Apr 2023'),'legacy',false) ...
+                struct('name','gpt-4-turbo-preview','attributes',struct('contextwindow',128000,'cutoff','Apr 2023'),'legacy',false), ...
+                struct('name','dall-e-3','attributes',struct('contextwindow','n/a','cutoff','n/a'),'legacy',false), ...
                 ];
             contextwindow = models(arrayfun(@(x) string(x.name), models) == modelName).attributes.contextwindow;
             cutoff = models(arrayfun(@(x) string(x.name), models) == modelName).attributes.cutoff;
@@ -37,7 +38,8 @@
                 "<ul><li>Runs on the <a href='https://github.com/matlab-deep-learning/llms-with-matlab' target='_blank'>LLMs with MATLAB</a> framework, which requires MATLAB R2023a or later</li>" + ...
                 "<li>Detects a URL included in a prompt, and retrieve its content into the chat.</<li>" + ...
                 "<li>Lets you import a .m, .mlx, .csv, or .txt file into the chat</li>" + ...
-                "<li>Supports GPT-4 Turbo with Vision with .jpg, .png or .gif images</li></ul>" + ...
+                "<li>Supports GPT-4 Turbo with Vision with .jpg, .png or .gif images</li>" + ...
+                "<li>Supports image generation via DALLe-3 API when streaming is disabled</li></ul>" + ...
                 "<strong>Notes</strong>:" + ...
                 "<ul><li>Imported content will be truncated if it exceeds the context window limit.</li>" + ...
                 "<li>PDF file is supported if Text Analytics Toolbox is available.</li></ul>"];
@@ -64,7 +66,7 @@
             args = "{""arg1"": 1 }";
             funCall = struct("name", functionName, "arguments", args);
             toolCall = struct("id", id, "type", "function", "function", funCall);
-            toolCallPrompt = struct("role", "assistant", "content", "", "tool_calls", toolCall);
+            toolCallPrompt = struct("role", "assistant", "content", [], "tool_calls", toolCall);
             messages = addResponseMessage(messages, toolCallPrompt);
             % add content as the function result
             messages = addToolMessage(messages,id,functionName,content);
@@ -74,7 +76,9 @@
             % remove test reports before sending user prompt    
 
             % extract content from messages
-            contents = cellfun(@(x) x.content, messages.Messages);
+            contents = strings(size(messages.Messages));
+            isText = ~cellfun(@(x) isempty(x.content), messages.Messages);
+            contents(isText) = cellfun(@(x) x.content, messages.Messages(isText));
             isTestReport = startsWith(contents,'<div class="test-report">');
             if any(isTestReport)
                 idx1 = find(isTestReport);
@@ -126,6 +130,18 @@
                     characterListPattern("./_-:;%&?#"), 1);
             url = extract(content,urlPat);
         end
-
+
+        % generate base64 encoded img tag
+        function imgTag = getHTMLImgTag(filepath)
+            [~,~,ext] = fileparts(filepath);
+            MIMEType = "data:image/" + erase(ext,".") + ";base64,";
+            fid = fopen(filepath);
+            byteArray = fread(fid,'*uint8');
+            fclose(fid);
+            b64char = matlab.net.base64encode(byteArray);
+            urlEncoded = MIMEType + b64char;
+            imgTag = "<img width=512 src=" + urlEncoded + ">";
+        end
+
     end
 end

helpers/llms-with-matlab/+llms/+internal/callOpenAIChatAPI.m

@@ -84,8 +84,20 @@
     if isempty(nvp.StreamFun)
         message = response.Body.Data.choices(1).message;
     else
-        message = struct("role", "assistant", ...
-            "content", streamedText);
+        pat = '{"' + wildcardPattern + '":';
+        if contains(streamedText,pat)
+            s = jsondecode(streamedText);
+            if contains(s.function.arguments,pat)
+                prompt = jsondecode(s.function.arguments);
+                s.function.arguments = prompt;
+            end
+            message = struct("role", "assistant", ...
+                 "content",[], ...
+                 "tool_calls",jsondecode(streamedText));
+        else
+            message = struct("role", "assistant", ...
+                "content", streamedText);
+        end
     end
     if isfield(message, "tool_choice")
         text = "";
@@ -131,7 +143,7 @@
 
 nvpOptions = keys(dict);
 if strcmp(nvp.ModelName,'gpt-4-vision-preview')
-    nvpOptions(ismember(nvpOptions,["MaxNumTokens","StopSequences"])) = [];
+    nvpOptions(ismember(nvpOptions,"StopSequences")) = [];
 end
 
 for opt = nvpOptions.'