+[](https://codespaces.new/geekan/MetaGPT)
-You may use the button above to open this repo in a Codespace
+Click the button above to open this repository in a Codespace. For additional information, refer to the [GitHub documentation on creating a Codespace](https://docs.github.com/en/free-pro-team@latest/github/developing-online-with-codespaces/creating-a-codespace#creating-a-codespace).
-For more info, check out the [GitHub documentation](https://docs.github.com/en/free-pro-team@latest/github/developing-online-with-codespaces/creating-a-codespace#creating-a-codespace).
-
## VS Code Dev Containers
-
+[](https://vscode.dev/redirect?url=vscode://ms-vscode-remote.remote-containers/cloneInVolume?url=https://github.com/geekan/MetaGPT)
-Note: If you click this link you will open the main repo and not your local cloned repo, you can use this link and replace with your username and cloned repo name:
-https://vscode.dev/redirect?url=vscode://ms-vscode-remote.remote-containers/cloneInVolume?url=https://github.com/geekan/MetaGPT
+Note: Clicking the link above opens the main repository. To open your local cloned repository, replace the URL with your username and cloned repository's name: `https://vscode.dev/redirect?url=vscode://ms-vscode-remote.remote-containers/cloneInVolume?url=https://github.com/
-
+
-Assign different roles to GPTs to form a collaborative software entity for complex tasks. +Assign different roles to GPTs to form a collaborative entity for complex tasks.
+
-
+
- 
Software Company Multi-Role Schematic (Gradually Implementing)
+## News +🚀 Oct. 29, 2024: We introduced three papers: [AFLOW](https://arxiv.org/abs/2410.10762), [FACT](https://arxiv.org/abs/2410.21012), and [SELA](https://arxiv.org/abs/2410.17238), check the [code](examples)! -## MetaGPT's Abilities +🚀 Mar. 29, 2024: [v0.8.0](https://github.com/geekan/MetaGPT/releases/tag/v0.8.0) released. Now you can use Data Interpreter ([arxiv](https://arxiv.org/abs/2402.18679), [example](https://docs.deepwisdom.ai/main/en/DataInterpreter/), [code](https://github.com/geekan/MetaGPT/tree/main/examples/di)) via pypi package import. Meanwhile, we integrated the RAG module and supported multiple new LLMs. +🚀 Feb. 08, 2024: [v0.7.0](https://github.com/geekan/MetaGPT/releases/tag/v0.7.0) released, supporting assigning different LLMs to different Roles. We also introduced [Data Interpreter](https://github.com/geekan/MetaGPT/blob/main/examples/di/README.md), a powerful agent capable of solving a wide range of real-world problems. -https://github.com/geekan/MetaGPT/assets/34952977/34345016-5d13-489d-b9f9-b82ace413419 +🚀 Jan. 16, 2024: Our paper [MetaGPT: Meta Programming for A Multi-Agent Collaborative Framework +](https://openreview.net/forum?id=VtmBAGCN7o) accepted for **oral presentation (top 1.2%)** at ICLR 2024, **ranking #1** in the LLM-based Agent category. +🚀 Jan. 03, 2024: [v0.6.0](https://github.com/geekan/MetaGPT/releases/tag/v0.6.0) released, new features include serialization, upgraded OpenAI package and supported multiple LLM, provided [minimal example for debate](https://github.com/geekan/MetaGPT/blob/main/examples/debate_simple.py) etc. +🚀 Dec. 15, 2023: [v0.5.0](https://github.com/geekan/MetaGPT/releases/tag/v0.5.0) released, introducing some experimental features such as incremental development, multilingual, multiple programming languages, etc. -## Examples (fully generated by GPT-4) +🔥 Nov. 08, 2023: MetaGPT is selected into [Open100: Top 100 Open Source achievements](https://www.benchcouncil.org/evaluation/opencs/annual.html). -For example, if you type `python startup.py "Design a RecSys like Toutiao"`, you would get many outputs, one of them is data & api design +🔥 Sep. 01, 2023: MetaGPT tops GitHub Trending Monthly for the **17th time** in August 2023. - +🌟 Jun. 30, 2023: MetaGPT is now open source. -It costs approximately **$0.2** (in GPT-4 API fees) to generate one example with analysis and design, and around **$2.0** for a full project. +🌟 Apr. 24, 2023: First line of MetaGPT code committed. +## Software Company as Multi-Agent System +1. MetaGPT takes a **one line requirement** as input and outputs **user stories / competitive analysis / requirements / data structures / APIs / documents, etc.** +2. Internally, MetaGPT includes **product managers / architects / project managers / engineers.** It provides the entire process of a **software company along with carefully orchestrated SOPs.** + 1. `Code = SOP(Team)` is the core philosophy. We materialize SOP and apply it to teams composed of LLMs. + -## Installation +Software Company Multi-Agent Schematic (Gradually Implementing)
-### Installation Video Guide +## Get Started -- [Matthew Berman: How To Install MetaGPT - Build A Startup With One Prompt!!](https://youtu.be/uT75J_KG_aY) +### Installation -### Traditional Installation +> Ensure that Python 3.9 or later, but less than 3.12, is installed on your system. You can check this by using: `python --version`. +> You can use conda like this: `conda create -n metagpt python=3.9 && conda activate metagpt` ```bash -# Step 1: Ensure that NPM is installed on your system. Then install mermaid-js. (If you don't have npm in your computer, please go to the Node.js offical website to install Node.js https://nodejs.org/ and then you will have npm tool in your computer.) -npm --version -sudo npm install -g @mermaid-js/mermaid-cli - -# Step 2: Ensure that Python 3.9+ is installed on your system. You can check this by using: -python --version - -# Step 3: Clone the repository to your local machine, and install it. -git clone https://github.com/geekan/metagpt -cd metagpt -pip install -e. +pip install --upgrade metagpt +# or `pip install --upgrade git+https://github.com/geekan/MetaGPT.git` +# or `git clone https://github.com/geekan/MetaGPT && cd MetaGPT && pip install --upgrade -e .` ``` -**Note:** - -- If already have Chrome, Chromium, or MS Edge installed, you can skip downloading Chromium by setting the environment variable - `PUPPETEER_SKIP_CHROMIUM_DOWNLOAD` to `true`. - -- Some people are [having issues](https://github.com/mermaidjs/mermaid.cli/issues/15) installing this tool globally. Installing it locally is an alternative solution, - - ```bash - npm install @mermaid-js/mermaid-cli - ``` - -- don't forget to the configuration for mmdc in config.yml - - ```yml - PUPPETEER_CONFIG: "./config/puppeteer-config.json" - MMDC: "./node_modules/.bin/mmdc" - ``` - -- if `pip install -e.` fails with error `[Errno 13] Permission denied: '/usr/local/lib/python3.11/dist-packages/test-easy-install-13129.write-test'`, try instead running `pip install -e. --user` - -- To convert Mermaid charts to SVG, PNG, and PDF formats. In addition to the Node.js version of Mermaid-CLI, you now have the option to use Python version Playwright, pyppeteer or mermaid.ink for this task. - - - Playwright - - **Install Playwright** - - ```bash - pip install playwright - ``` - - - **Install the Required Browsers** - - to support PDF conversion, please install Chrominum. - - ```bash - playwright install --with-deps chromium - ``` - - - **modify `config.yaml`** - - uncomment MERMAID_ENGINE from config.yaml and change it to `playwright` - - ```yaml - MERMAID_ENGINE: playwright - ``` - - - pyppeteer - - **Install pyppeteer** - - ```bash - pip install pyppeteer - ``` - - - **Use your own Browsers** - - pyppeteer alow you use installed browsers, please set the following envirment - - ```bash - export PUPPETEER_EXECUTABLE_PATH = /path/to/your/chromium or edge or chrome - ``` - - please do not use this command to install browser, it is too old - - ```bash - pyppeteer-install - ``` - - - **modify `config.yaml`** - - uncomment MERMAID_ENGINE from config.yaml and change it to `pyppeteer` +For detailed installation guidance, please refer to [cli_install](https://docs.deepwisdom.ai/main/en/guide/get_started/installation.html#install-stable-version) + or [docker_install](https://docs.deepwisdom.ai/main/en/guide/get_started/installation.html#install-with-docker) - ```yaml - MERMAID_ENGINE: pyppeteer - ``` - - - mermaid.ink - - **modify `config.yaml`** - - uncomment MERMAID_ENGINE from config.yaml and change it to `ink` - - ```yaml - MERMAID_ENGINE: ink - ``` - - Note: this method does not support pdf export. - -### Installation by Docker +### Configuration +You can init the config of MetaGPT by running the following command, or manually create `~/.metagpt/config2.yaml` file: ```bash -# Step 1: Download metagpt official image and prepare config.yaml -docker pull metagpt/metagpt:v0.3.1 -mkdir -p /opt/metagpt/{config,workspace} -docker run --rm metagpt/metagpt:v0.3.1 cat /app/metagpt/config/config.yaml > /opt/metagpt/config/key.yaml -vim /opt/metagpt/config/key.yaml # Change the config - -# Step 2: Run metagpt demo with container -docker run --rm \ - --privileged \ - -v /opt/metagpt/config/key.yaml:/app/metagpt/config/key.yaml \ - -v /opt/metagpt/workspace:/app/metagpt/workspace \ - metagpt/metagpt:v0.3.1 \ - python startup.py "Write a cli snake game" - -# You can also start a container and execute commands in it -docker run --name metagpt -d \ - --privileged \ - -v /opt/metagpt/config/key.yaml:/app/metagpt/config/key.yaml \ - -v /opt/metagpt/workspace:/app/metagpt/workspace \ - metagpt/metagpt:v0.3.1 - -docker exec -it metagpt /bin/bash -$ python startup.py "Write a cli snake game" +# Check https://docs.deepwisdom.ai/main/en/guide/get_started/configuration.html for more details +metagpt --init-config # it will create ~/.metagpt/config2.yaml, just modify it to your needs ``` -The command `docker run ...` do the following things: +You can configure `~/.metagpt/config2.yaml` according to the [example](https://github.com/geekan/MetaGPT/blob/main/config/config2.example.yaml) and [doc](https://docs.deepwisdom.ai/main/en/guide/get_started/configuration.html): -- Run in privileged mode to have permission to run the browser -- Map host directory `/opt/metagpt/config` to container directory `/app/metagpt/config` -- Map host directory `/opt/metagpt/workspace` to container directory `/app/metagpt/workspace` -- Execute the demo command `python startup.py "Write a cli snake game"` - -### Build image by yourself - -```bash -# You can also build metagpt image by yourself. -git clone https://github.com/geekan/MetaGPT.git -cd MetaGPT && docker build -t metagpt:custom . +```yaml +llm: + api_type: "openai" # or azure / ollama / groq etc. Check LLMType for more options + model: "gpt-4-turbo" # or gpt-3.5-turbo + base_url: "https://api.openai.com/v1" # or forward url / other llm url + api_key: "YOUR_API_KEY" ``` -## Configuration +### Usage -- Configure your `OPENAI_API_KEY` in any of `config/key.yaml / config/config.yaml / env` -- Priority order: `config/key.yaml > config/config.yaml > env` +After installation, you can use MetaGPT at CLI ```bash -# Copy the configuration file and make the necessary modifications. -cp config/config.yaml config/key.yaml +metagpt "Create a 2048 game" # this will create a repo in ./workspace ``` -| Variable Name | config/key.yaml | env | -| ------------------------------------------ | ----------------------------------------- | ----------------------------------------------- | -| OPENAI_API_KEY # Replace with your own key | OPENAI_API_KEY: "sk-..." | export OPENAI_API_KEY="sk-..." | -| OPENAI_API_BASE # Optional | OPENAI_API_BASE: "https://
-
+
@@ -9,20 +9,20 @@
-
-
-
-
+
+
+
+
-
-
软件公司多角色示意图（正在逐步实现）
-## MetaGPT 的能力 - -https://github.com/geekan/MetaGPT/assets/34952977/34345016-5d13-489d-b9f9-b82ace413419 - - -## 示例（均由 GPT-4 生成） - -例如，键入`python startup.py "写个类似今日头条的推荐系统"`并回车，你会获得一系列输出，其一是数据结构与API设计 - - - -这需要大约**0.2美元**（GPT-4 API的费用）来生成一个带有分析和设计的示例，大约2.0美元用于一个完整的项目 - ## 安装 +### Pip安装 -### 传统安装 +> 确保您的系统已安装 Python 3.9 或更高版本。您可以使用以下命令来检查：`python --version`。 +> 您可以这样使用 conda：`conda create -n metagpt python=3.9 && conda activate metagpt` ```bash -# 第 1 步：确保您的系统上安装了 NPM。并使用npm安装mermaid-js -npm --version -sudo npm install -g @mermaid-js/mermaid-cli - -# 第 2 步：确保您的系统上安装了 Python 3.9+。您可以使用以下命令进行检查： -python --version - -# 第 3 步：克隆仓库到您的本地机器，并进行安装。 -git clone https://github.com/geekan/metagpt -cd metagpt -pip install -e. +pip install metagpt +metagpt --init-config # 创建 ~/.metagpt/config2.yaml，根据您的需求修改它 +metagpt "创建一个 2048 游戏" # 这将在 ./workspace 创建一个仓库 ``` -**注意：** - -- 如果已经安装了Chrome、Chromium或MS Edge，可以通过将环境变量`PUPPETEER_SKIP_CHROMIUM_DOWNLOAD`设置为`true`来跳过下载Chromium。 - -- 一些人在全局安装此工具时遇到问题。在本地安装是替代解决方案， - - ```bash - npm install @mermaid-js/mermaid-cli - ``` +或者您可以将其作为库使用 -- 不要忘记在config.yml中为mmdc配置配置， - - ```yml - PUPPETEER_CONFIG: "./config/puppeteer-config.json" - MMDC: "./node_modules/.bin/mmdc" - ``` +```python +from metagpt.software_company import generate_repo, ProjectRepo +repo: ProjectRepo = generate_repo("创建一个 2048 游戏") # 或 ProjectRepo("<路径>") +print(repo) # 它将打印出仓库结构及其文件 +``` -- 如果`pip install -e.`失败并显示错误`[Errno 13] Permission denied: '/usr/local/lib/python3.11/dist-packages/test-easy-install-13129.write-test'`，请尝试使用`pip install -e. --user`运行。 +详细的安装请参考 [cli_install](https://docs.deepwisdom.ai/guide/get_started/installation.html#install-stable-version) ### Docker安装 +> 注意：在Windows中，你需要将 "/opt/metagpt" 替换为Docker具有创建权限的目录，比如"D:\Users\x\metagpt" ```bash -# 步骤1: 下载metagpt官方镜像并准备好config.yaml -docker pull metagpt/metagpt:v0.3 +# 步骤1: 下载metagpt官方镜像并准备好config2.yaml +docker pull metagpt/metagpt:latest mkdir -p /opt/metagpt/{config,workspace} -docker run --rm metagpt/metagpt:v0.3 cat /app/metagpt/config/config.yaml > /opt/metagpt/config/config.yaml -vim /opt/metagpt/config/config.yaml # 修改config +docker run --rm metagpt/metagpt:latest cat /app/metagpt/config/config2.yaml > /opt/metagpt/config/config2.yaml +vim /opt/metagpt/config/config2.yaml # 修改配置文件 # 步骤2: 使用容器运行metagpt演示 docker run --rm \ --privileged \ - -v /opt/metagpt/config:/app/metagpt/config \ - -v /opt/metagpt/workspace:/app/metagpt/workspace \ - metagpt/metagpt:v0.3 \ - python startup.py "Write a cli snake game" - -# 您也可以启动一个容器并在其中执行命令 -docker run --name metagpt -d \ - --privileged \ - -v /opt/metagpt/config:/app/metagpt/config \ + -v /opt/metagpt/config/config2.yaml:/app/metagpt/config/config2.yaml \ -v /opt/metagpt/workspace:/app/metagpt/workspace \ - metagpt/metagpt:v0.3 - -docker exec -it metagpt /bin/bash -$ python startup.py "Write a cli snake game" -``` - -`docker run ...`做了以下事情: - -- 以特权模式运行，有权限运行浏览器 -- 将主机目录 `/opt/metagpt/config` 映射到容器目录`/app/metagpt/config` -- 将主机目录 `/opt/metagpt/workspace` 映射到容器目录 `/app/metagpt/workspace` -- 执行演示命令 `python startup.py "Write a cli snake game"` - -### 自己构建镜像 - -```bash -# 您也可以自己构建metagpt镜像 -git clone https://github.com/geekan/MetaGPT.git -cd MetaGPT && docker build -t metagpt:v0.3 . + metagpt/metagpt:latest \ + metagpt "Write a cli snake game" ``` -## 配置 - -- 在 `config/key.yaml / config/config.yaml / env` 中配置您的 `OPENAI_API_KEY` -- 优先级顺序：`config/key.yaml > config/config.yaml > env` +详细的安装请参考 [docker_install](https://docs.deepwisdom.ai/main/zh/guide/get_started/installation.html#%E4%BD%BF%E7%94%A8docker%E5%AE%89%E8%A3%85) -```bash -# 复制配置文件并进行必要的修改 -cp config/config.yaml config/key.yaml -``` +### 快速开始的演示视频 +- 在 [MetaGPT Huggingface Space](https://huggingface.co/spaces/deepwisdom/MetaGPT-SoftwareCompany) 上进行体验 +- [Matthew Berman: How To Install MetaGPT - Build A Startup With One Prompt!!](https://youtu.be/uT75J_KG_aY) +- [官方演示视频](https://github.com/geekan/MetaGPT/assets/2707039/5e8c1062-8c35-440f-bb20-2b0320f8d27d) -| 变量名 | config/key.yaml | env | -| ----------------------------------- | ----------------------------------------- | ----------------------------------------------- | -| OPENAI_API_KEY # 用您自己的密钥替换 | OPENAI_API_KEY: "sk-..." | export OPENAI_API_KEY="sk-..." | -| OPENAI_API_BASE # 可选 | OPENAI_API_BASE: "https://
+
+
+Assigner différents rôles aux GPTs pour former une entité collaborative capable de gérer des tâches complexes. +
+ +
+
+
+
+
+
+
+
+
Schéma multi-agent d'une entreprise de logiciels (Mise en œuvre progressive)
+ + +## Commençons ! + +### Installation + +> Assurez-vous que Python 3.9 ou supérieur, mais inférieur à 3.12, est installé sur votre système. Vous pouvez le vérifier en utilisant : `python --version`. +> Vous pouvez utiliser conda comme suit : `conda create -n metagpt python=3.9 && conda activate metagpt` + +```bash +pip install --upgrade metagpt +# or `pip install --upgrade git+https://github.com/geekan/MetaGPT.git` +# or `git clone https://github.com/geekan/MetaGPT && cd MetaGPT && pip install --upgrade -e .` +``` + +Pour des conseils d'installation détaillés, veuillez vous référer à [cli_install](https://docs.deepwisdom.ai/main/en/guide/get_started/installation.html#install-stable-version) + ou [docker_install](https://docs.deepwisdom.ai/main/en/guide/get_started/installation.html#install-with-docker) + +### Configuration + +Vous pouvez initialiser la configuration de MetaGPT en lançant la commande suivante, ou en créant manuellement le fichier `~/.metagpt/config2.yaml` : +```bash +# Visitez https://docs.deepwisdom.ai/main/en/guide/get_started/configuration.html pour plus de détails +metagpt --init-config # il créera ~/.metagpt/config2.yaml, il suffit de le modifier selon vos besoins +``` + +Vous pouvez configurer `~/.metagpt/config2.yaml` selon l'[exemple](https://github.com/geekan/MetaGPT/blob/main/config/config2.example.yaml) et le [doc](https://docs.deepwisdom.ai/main/en/guide/get_started/configuration.html) : + +```yaml +llm: + api_type: "openai" # ou azure / ollama / groq etc. Consultez LLMType pour plus d'options + model: "gpt-4-turbo" # ou gpt-3.5-turbo + base_url: "https://api.openai.com/v1" # ou URL de transfert / URL d'autre LLM. + api_key: "VOTRE_CLE_API" +``` + +### Utilisation + +Après l'installation, vous pouvez utiliser MetaGPT en CLI + +```bash +metagpt "Create a 2048 game" # ceci créera un repo dans ./workspace +``` + +ou l'utiliser comme bibliothèque + +```python +from metagpt.software_company import generate_repo, ProjectRepo +repo: ProjectRepo = generate_repo("Create a 2048 game") # ou ProjectRepo("
-
+
@@ -9,20 +9,20 @@
-
-
-
+
+
+
+
-
ソフトウェア会社のマルチロール図式（順次導入）
-## MetaGPTの能力 +## MetaGPT の能力 + https://github.com/geekan/MetaGPT/assets/34952977/34345016-5d13-489d-b9f9-b82ace413419 + ## 例（GPT-4 で完全生成） -例えば、`python startup.py "Toutiao のような RecSys をデザインする"`と入力すると、多くの出力が得られます +例えば、`metagpt "Toutiao のような RecSys をデザインする"`と入力すると、多くの出力が得られます  解析と設計を含む 1 つの例を生成するのに約 **$0.2**（GPT-4 の API 使用料）、完全なプロジェクトでは約 **$2.0** かかります。 + + + ## インストール ### インストールビデオガイド @@ -53,19 +58,21 @@ https://github.com/geekan/MetaGPT/assets/34952977/34345016-5d13-489d-b9f9-b82ace - [Matthew Berman: How To Install MetaGPT - Build A Startup With One Prompt!!](https://youtu.be/uT75J_KG_aY) ### 伝統的なインストール +> Python 3.9 以上がシステムにインストールされていることを確認してください。これは `python --version` を使ってチェックできます。 +> 以下のようにcondaを使うことができます：`conda create -n metagpt python=3.9 && conda activate metagpt` ```bash -# ステップ 1: NPM がシステムにインストールされていることを確認してください。次に mermaid-js をインストールします。 -npm --version -sudo npm install -g @mermaid-js/mermaid-cli +pip install metagpt +metagpt --init-config # ~/.metagpt/config2.yaml を作成し、自分の設定に合わせて変更してください +metagpt "2048ゲームを作成する" # これにより ./workspace にリポジトリが作成されます +``` -# ステップ 2: Python 3.9+ がシステムにインストールされていることを確認してください。これを確認するには: -python --version +または、ライブラリとして使用することもできます -# ステップ 3: リポジトリをローカルマシンにクローンし、インストールする。 -git clone https://github.com/geekan/metagpt -cd metagpt -pip install -e. +```python +from metagpt.software_company import generate_repo, ProjectRepo +repo: ProjectRepo = generate_repo("2048ゲームを作成する") # または ProjectRepo("<パス>") +print(repo) # リポジトリの構造とファイルを出力します ``` **注:** @@ -79,49 +86,118 @@ Chromium のダウンロードをスキップすることができます。 npm install @mermaid-js/mermaid-cli ``` -- config.yml に mmdc のコンフィギュレーションを記述するのを忘れないこと +- config.yml に mmdc のコンフィグを記述するのを忘れないこと ```yml - PUPPETEER_CONFIG: "./config/puppeteer-config.json" - MMDC: "./node_modules/.bin/mmdc" + puppeteer_config: "./config/puppeteer-config.json" + path: "./node_modules/.bin/mmdc" ``` - もし `pip install -e.` がエラー `[Errno 13] Permission denied: '/usr/local/lib/python3.11/dist-packages/test-easy-install-13129.write-test'` で失敗したら、代わりに `pip install -e. --user` を実行してみてください +- Mermaid charts を SVG、PNG、PDF 形式に変換します。Node.js 版の Mermaid-CLI に加えて、Python 版の Playwright、pyppeteer、または mermaid.ink をこのタスクに使用できるようになりました。 + + - Playwright + - **Playwright のインストール** + + ```bash + pip install playwright + ``` + + - **必要なブラウザのインストール** + + PDF変換をサポートするには、Chrominumをインストールしてください。 + + ```bash + playwright install --with-deps chromium + ``` + + - **modify `config2.yaml`** + + config2.yaml から mermaid.engine のコメントを外し、`playwright` に変更する + + ```yaml + mermaid: + engine: playwright + ``` + + - pyppeteer + - **pyppeteer のインストール** + + ```bash + pip install pyppeteer + ``` + + - **自分のブラウザを使用** + + pyppeteer を使えばインストールされているブラウザを使うことができます、以下の環境を設定してください + + ```bash + export PUPPETEER_EXECUTABLE_PATH = /path/to/your/chromium or edge or chrome + ``` + + ブラウザのインストールにこのコマンドを使わないでください、これは古すぎます + + ```bash + pyppeteer-install + ``` + + - **`config2.yaml` を修正** + + config2.yaml から mermaid.engine のコメントを外し、`pyppeteer` に変更する + + ```yaml + mermaid: + engine: pyppeteer + ``` + + - mermaid.ink + - **`config2.yaml` を修正** + + config2.yaml から mermaid.engine のコメントを外し、`ink` に変更する + + ```yaml + mermaid: + engine: ink + ``` + + 注: この方法は pdf エクスポートに対応していません。 + ### Docker によるインストール +> Windowsでは、"/opt/metagpt"をDockerが作成する権限を持つディレクトリに置き換える必要があります。例えば、"D:\Users\x\metagpt"などです。 ```bash -# ステップ 1: metagpt 公式イメージをダウンロードし、config.yaml を準備する -docker pull metagpt/metagpt:v0.3.1 +# ステップ 1: metagpt 公式イメージをダウンロードし、config2.yaml を準備する +docker pull metagpt/metagpt:latest mkdir -p /opt/metagpt/{config,workspace} -docker run --rm metagpt/metagpt:v0.3.1 cat /app/metagpt/config/config.yaml > /opt/metagpt/config/key.yaml -vim /opt/metagpt/config/key.yaml # 設定を変更する +docker run --rm metagpt/metagpt:latest cat /app/metagpt/config/config2.yaml > /opt/metagpt/config/config2.yaml +vim /opt/metagpt/config/config2.yaml # 設定を変更する # ステップ 2: コンテナで metagpt デモを実行する docker run --rm \ --privileged \ - -v /opt/metagpt/config/key.yaml:/app/metagpt/config/key.yaml \ + -v /opt/metagpt/config/config2.yaml:/app/metagpt/config/config2.yaml \ -v /opt/metagpt/workspace:/app/metagpt/workspace \ - metagpt/metagpt:v0.3.1 \ - python startup.py "Write a cli snake game" + metagpt/metagpt:latest \ + metagpt "Write a cli snake game" # コンテナを起動し、その中でコマンドを実行することもできます docker run --name metagpt -d \ --privileged \ - -v /opt/metagpt/config/key.yaml:/app/metagpt/config/key.yaml \ + -v /opt/metagpt/config/config2.yaml:/app/metagpt/config/config2.yaml \ -v /opt/metagpt/workspace:/app/metagpt/workspace \ - metagpt/metagpt:v0.3.1 + metagpt/metagpt:latest docker exec -it metagpt /bin/bash -$ python startup.py "Write a cli snake game" +$ metagpt "Write a cli snake game" ``` コマンド `docker run ...` は以下のことを行います: - 特権モードで実行し、ブラウザの実行権限を得る -- ホストディレクトリ `/opt/metagpt/config` をコンテナディレクトリ `/app/metagpt/config` にマップする -- ホストディレクトリ `/opt/metagpt/workspace` をコンテナディレクトリ `/app/metagpt/workspace` にマップする -- デモコマンド `python startup.py "Write a cli snake game"` を実行する +- ホスト設定ファイル `/opt/metagpt/config/config2.yaml` をコンテナ `/app/metagpt/config/config2.yaml` にマップします +- ホストディレクトリ `/opt/metagpt/workspace` をコンテナディレクトリ `/app/metagpt/workspace` にマップするs +- デモコマンド `metagpt "Write a cli snake game"` を実行する ### 自分でイメージをビルドする @@ -133,28 +209,23 @@ cd MetaGPT && docker build -t metagpt:custom . ## 設定 -- `OPENAI_API_KEY` を `config/key.yaml / config/config.yaml / env` のいずれかで設定します。 -- 優先順位は: `config/key.yaml > config/config.yaml > env` の順です。 +- `api_key` を `~/.metagpt/config2.yaml / config/config2.yaml` のいずれかで設定します。 +- 優先順位は: `~/.metagpt/config2.yaml > config/config2.yaml > env` の順です。 ```bash # 設定ファイルをコピーし、必要な修正を加える。 -cp config/config.yaml config/key.yaml +cp config/config2.yaml ~/.metagpt/config2.yaml ``` -| 変数名 | config/key.yaml | env | -| --------------------------------------- | ----------------------------------------- | ----------------------------------------------- | -| OPENAI_API_KEY # 自分のキーに置き換える | OPENAI_API_KEY: "sk-..." | export OPENAI_API_KEY="sk-..." | -| OPENAI_API_BASE # オプション | OPENAI_API_BASE: "https://
+
+
+
+
+
+
A small utility plugin for allowing users to collect blocks using a higher level API.
+ +
+ 
+ 
+
+After remembering the location where you want to operate, a request similar to the one below will be output in the terminal. Reply to it and thereby direct the Android assistant to learn your demonstration action:
+
+```bash
+| INFO | examples.android_assistant.actions.manual_record:run:96 - Which element do you want to tap? Choose a numeric tag from 1 to 11:
+user_input: 8
+| INFO | examples.android_assistant.actions.manual_record:run:81 - Choose one of the following actions you want to perform on the current screen:
+tap, text, long_press, swipe, stop
+user_input: tap
+```
+
+### Automatic Execution Stage
+After the Android Assistant completes the learning stage, you can command it to complete tasks on the phone through text descriptions. By configuring the operation documents from the self-learning stage, the Android Assistant has richer prior knowledge, and its execution capabilities are further enhanced.
+You can instruct the Android Assistant to send messages in the "Messenger" app with the following command:
+```bash
+python run_assistant.py "Send 'When will we release this feature?' to +86 8888888" --stage "act" --mode "auto or manual" --app-name "Messenger"
+```
+Specifically, by selecting `auto` for `mode`, the Android assistant will employ the operational records compiled through self-exploration. Alternatively, if `manual` is chosen as the `mode`, the Android assistant will leverage the operation manuals accrued from learning via human demonstration.
+
+## Installation
+To use the Android Assistant, you first need to meet the following conditions:
+1. Complete the installation of the MetaGPT environment.
+2. Install [Android Debug Bridge (ADB)](https://developer.android.com/tools/adb?hl=zh-cn) on your PC, which enables interaction between your PC and Android devices.
+3. Install Android Studio and within it, install the Android emulator to provide an environment for the Android Assistant to learn and execute. For information on how to install the Android emulator, refer to [Quick Installation of Android Studio & Emulator](https://docs.expo.dev/workflow/android-studio-emulator/).
+4. (Optional) Connect your Android device to the USB port of your PC, which can also provide an environment for the Android Assistant to learn and execute.
+
+Note ⚠️: When operating with the Android emulator, the emulator model we use is Medium Phone, which is recommended for first-time users to complete the operation.
+
+After completing these operations, you can enter the following command to check if ADB is installed successfully and if the Android device is connected:
+```bash
+adb devices
+```
+
+## Usage
+The MetaGPT Android Assistant is designed within the MetaGPT framework as a collection of Roles and multiple Actions. You can run it by executing the `run_assistant.py` script. The specific parameter description of this script is as follows:
+```text
+Usage: run_assistant.py [OPTIONS] TASK_DESC
+
+ Run a Android Assistant
+
+Arguments:
+ TASK_DESC the task description you want the android assistant to learn or
+ act [required]
+
+Options:
+ --n-round INTEGER The max round to do an app operation task.
+ [default: 20]
+ --stage TEXT stage: learn / act [default: learn]
+ --mode TEXT mode: auto / manual , when state=learn
+ [default: auto]
+ --app-name TEXT the name of app you want to run [default:
+ demo]
+ --investment FLOAT Dollar amount to invest in the AI company.
+ [default: 5.0]
+ --refine-doc / --no-refine-doc Refine existing operation docs based on the
+ latest observation if True. [default: no-
+ refine-doc]
+ --min-dist INTEGER The minimum distance between elements to
+ prevent overlapping during the labeling
+ process. [default: 30]
+ --android-screenshot-dir TEXT The path to store screenshots on android
+ device. Make sure it exists. [default:
+ /sdcard/Pictures/Screenshots]
+ --android-xml-dir TEXT The path to store xml files for determining
+ UI elements localtion. Make sure it exists.
+ [default: /sdcard]
+ --device-id TEXT The Android device_id [default:
+ emulator-5554]
+ --help Show this message and exit.
+```
+
+## Acknowledgements
+The MetaGPT Android Assistant has referenced some ideas and code from the [AppAgent](https://github.com/mnotgod96/AppAgent) project. We thank the developers of the Appagent project.
+
+### Citation
+
+```bib
+@misc{yang2023appagent,
+ title={AppAgent: Multimodal Agents as Smartphone Users},
+ author={Chi Zhang and Zhao Yang and Jiaxuan Liu and Yucheng Han and Xin Chen and Zebiao Huang and Bin Fu and Gang Yu},
+ year={2023},
+ eprint={2312.13771},
+ archivePrefix={arXiv},
+ primaryClass={cs.CV}
+}
+```
\ No newline at end of file
diff --git a/metagpt/ext/android_assistant/README_CN.md b/metagpt/ext/android_assistant/README_CN.md
new file mode 100644
index 000000000..a1abbe3b0
--- /dev/null
+++ b/metagpt/ext/android_assistant/README_CN.md
@@ -0,0 +1,113 @@
+# MetaGPT 安卓助理
+
+MetaGPT安卓助理是一款依托于先进的MetaGPT框架构建的多模态大语言模型驱动的智能辅助工具。
+它具备自我学习的能力，能够通过学习掌握用户的日常使用方式，同时能够根据用户的指令自动完成各类应用程序的操作任务，实现了用户双手的全面解放。
+接下来，我们将介绍MetaGPT安卓助理的功能以及如何使用它。
+
+## 功能
+
+MetaGPT 安卓助理的执行主要包含两个阶段，分别为自我学习与自动执行。下面，我们将从这两个阶段介绍MetaGPT 安卓助理的具体功能。
+
+### 自我学习阶段
+
+通过学习人类演示或基于人类指令对app进行探索，MetaGPT安卓助理可以对app的功能进行学习，生成相应的操作文档，为后续的“自动执行”阶段使用。对于任何给定的任务目标，进行约20轮的探索可以显著提高性能。
+
+通过设定`stage`为`learn`可要求安卓助理进入自我学习阶段。通过设定`mode`为`auto`，可要求安卓助理通过自动探索学习，通过设定`mode`为`manual`，可要求安卓助理通过人类手动演示学习。在使用章节，我们对脚本的参数进行了详细的说明。
+您可以尝试对“Messenger”应用程序进行自动探索和手动演示模式的实验，具体命令如下：
+
+```bash
+cd examples/android_assistant
+python run_assistant.py "Send 'When will we release this feature? to +86 8888888'" --stage "learn" --mode "auto or manual" --app-name "Messenger"
+```
+
+#### 基于人类演示的学习
+在要求安卓助理在自我学习阶段执行自我探索时，您可以解放您的双手，但在要求他根据您的指令进行学习时，你需要根据终端中的指令进行输入，以便安卓助理能够准确地学习您的操作方式。
+一个可能的例子如下：
+
+```bash
+cd examples/android_assistant
+python run_assistant.py "Send 'When will we release this feature? to +86 8888888'" --stage "learn" --mode "manual" --app-name "Messenger"
+```
+
+在运行这一指令后，你将首先看到一个在各个可交互的位置进行了标记的安卓屏幕的截图，如下图：
+
+
+
+在记住你要操作的位置之后，终端中将会输出与下面类似的要求，回复它，进而指挥安卓助理学习你的演示行为：
+
+```bash
+| INFO | examples.android_assistant.actions.manual_record:run:96 - Which element do you want to tap? Choose a numeric tag from 1 to 11:
+user_input: 8
+| INFO | examples.android_assistant.actions.manual_record:run:81 - Choose one of the following actions you want to perform on the current screen:
+tap, text, long_press, swipe, stop
+user_input: tap
+```
+### 自动执行阶段
+在安卓助理完成了自我学习阶段之后，您可以通过文本描述的方式，指挥安卓助理在手机中完成任务。通过为其配置自我学习阶段的操作文档，安卓助理具备了更丰富的前置知识，执行能力进一步得到提升。
+你可以通过以下指令，指挥安卓助理在“Messenger”应用中发送信息：
+```bash
+python run_assistant.py "Send 'When will we release this feature? to +86 8888888'" --stage "act" --mode "auto or manual" --app-name "Messenger"
+```
+其中，`mode`选择`auto`，安卓助理将使用自我探索中积累的操作文档；`mode`选择`manual`，安卓助理将使用人类演示学习中积累的操作文档。
+
+## 安装
+为了使用安卓助理，你首先需要满足以下条件：
+1. 完成MetaGPT环境的安装
+2. 在你的PC上安装[Android Debug Bridge(ADB)](https://developer.android.com/tools/adb?hl=zh-cn)，ADB可以使你的PC与安卓设备进行交互。
+3. 安装Android Studio，在其中安装Android模拟器，以为安卓助手提供学习与执行的环境。关于如何安装Android模拟器，可以参考[快速安装Android Studio & Emulator](https://dev.weixin.qq.com/docs/framework/dev/framework/env/android-simulator.html)。
+4. (Optional) 将你的安卓设备连接到PC的USB端口上，这同样可以为安卓助手提供学习与执行的环境。
+
+注意 ⚠️：在使用Android模拟器进行操作时，我们使用的模拟器型号为Medium Phone，建议第一次尝试此类应用的用户使用这一型号完成操作。
+
+在完成这一系列操作之后，你可以输入以下命令检查ADB是否安装成功，以及安卓设备是否连接
+```bash
+adb devices
+```
+## 使用
+MetaGPT 安卓助理在MetaGPT框架中被设计为一个`Role`与多个`Action`的集合，你可以通过运行`run_assistant.py`脚本来运行它。这一脚本具体的参数说明如下：
+```text
+用法：run_assistant.py [选项] 任务描述
+
+ 运行一个安卓助手
+
+参数：
+ TASK_DESC 你希望安卓助手学习或执行的任务描述
+ [必需]
+
+选项：
+ --n-round 整数 执行应用程序操作任务的最大轮数。
+ [默认值：20]
+ --stage 文本 阶段：learn/act [默认值：learn]
+ --mode 文本 模式：auto/manual，当状态=learn时 [默认值：auto]
+ --app-name 文本 你想要运行的应用程序名称 [默认值：
+ 演示]
+ --investment 浮点数 投资于人工智能公司的美元金额。
+ [默认值：5.0]
+ --refine-doc / --no-refine-doc 如果为真，则根据最新的观察结果优化现有操作文档。
+ [默认值：--no-refine-doc]
+ --min-dist 整数 在标记过程中防止元素重叠的最小元素间距。
+ [默认值：30]
+ --android-screenshot-dir 文本 在安卓设备上存储截图的路径。确保其存在。
+ [默认值：/sdcard/Pictures/Screenshots]
+ --android-xml-dir 文本 存储用于确定UI元素位置的XML文件的路径。
+ 确保其存在。[默认值：/sdcard]
+ --device-id 文本 安卓device_id [默认值：
+ 模拟器-5554]
+ --help 显示此信息并退出。
+```
+
+## 致谢
+MetaGPT 安卓助理参考了 [AppAgent](https://github.com/mnotgod96/AppAgent) 项目的部分思路与代码，感谢 Appagent 项目的开发者们。
+
+### 引用
+
+```bib
+@misc{yang2023appagent,
+ title={AppAgent: Multimodal Agents as Smartphone Users},
+ author={Chi Zhang and Zhao Yang and Jiaxuan Liu and Yucheng Han and Xin Chen and Zebiao Huang and Bin Fu and Gang Yu},
+ year={2023},
+ eprint={2312.13771},
+ archivePrefix={arXiv},
+ primaryClass={cs.CV}
+}
+```
\ No newline at end of file
diff --git a/metagpt/ext/android_assistant/__init__.py b/metagpt/ext/android_assistant/__init__.py
new file mode 100644
index 000000000..2bcf8efd0
--- /dev/null
+++ b/metagpt/ext/android_assistant/__init__.py
@@ -0,0 +1,3 @@
+#!/usr/bin/env python
+# -*- coding: utf-8 -*-
+# @Desc :
diff --git a/metagpt/ext/android_assistant/actions/__init__.py b/metagpt/ext/android_assistant/actions/__init__.py
new file mode 100644
index 000000000..2bcf8efd0
--- /dev/null
+++ b/metagpt/ext/android_assistant/actions/__init__.py
@@ -0,0 +1,3 @@
+#!/usr/bin/env python
+# -*- coding: utf-8 -*-
+# @Desc :
diff --git a/metagpt/ext/android_assistant/actions/manual_record.py b/metagpt/ext/android_assistant/actions/manual_record.py
new file mode 100644
index 000000000..bcfb2ed89
--- /dev/null
+++ b/metagpt/ext/android_assistant/actions/manual_record.py
@@ -0,0 +1,168 @@
+#!/usr/bin/env python
+# -*- coding: utf-8 -*-
+# @Desc : manual record user interaction in stage=learn & mode=manual, LIKE scripts/step_recorder.py
+import time
+from pathlib import Path
+
+import cv2
+
+from metagpt.actions.action import Action
+from metagpt.config2 import config
+from metagpt.environment.android.android_env import AndroidEnv
+from metagpt.environment.android.const import ADB_EXEC_FAIL
+from metagpt.environment.android.env_space import (
+ EnvAction,
+ EnvActionType,
+ EnvObsParams,
+ EnvObsType,
+)
+from metagpt.ext.android_assistant.utils.schema import (
+ ActionOp,
+ AndroidActionOutput,
+ RunState,
+ SwipeOp,
+)
+from metagpt.ext.android_assistant.utils.utils import (
+ draw_bbox_multi,
+ elem_list_from_xml_tree,
+)
+from metagpt.logs import logger
+
+
+class ManualRecord(Action):
+ """do a human operation on the screen with human input"""
+
+ name: str = "ManualRecord"
+
+ useless_list: list[str] = [] # store useless elements uid
+ record_path: Path = ""
+ task_desc_path: Path = ""
+ screenshot_before_path: Path = ""
+ screenshot_after_path: Path = ""
+ xml_path: Path = ""
+
+ async def run(self, task_desc: str, task_dir: Path, env: AndroidEnv):
+ self.record_path = Path(task_dir) / "record.txt"
+ self.task_desc_path = Path(task_dir) / "task_desc.txt"
+ self.screenshot_before_path = Path(task_dir) / "raw_screenshots"
+ self.screenshot_after_path = Path(task_dir) / "labeled_screenshots"
+ self.xml_path = Path(task_dir) / "xml"
+ for path in [self.screenshot_before_path, self.screenshot_after_path, self.xml_path]:
+ path.mkdir(parents=True, exist_ok=True)
+
+ self.record_path.write_text("")
+ record_file = open(self.record_path, "w")
+ self.task_desc_path.write_text(task_desc)
+
+ step = 0
+ extra_config = config.extra
+ while True:
+ step += 1
+ screenshot_path: Path = env.observe(
+ EnvObsParams(
+ obs_type=EnvObsType.GET_SCREENSHOT, ss_name=f"{step}", local_save_dir=self.screenshot_before_path
+ )
+ )
+ xml_path: Path = env.observe(
+ EnvObsParams(obs_type=EnvObsType.GET_XML, xml_name=f"{step}", local_save_dir=self.xml_path)
+ )
+ if not screenshot_path.exists() or not xml_path.exists():
+ return AndroidActionOutput(action_state=RunState.FAIL)
+
+ elem_list = elem_list_from_xml_tree(xml_path, self.useless_list, extra_config.get("min_dist", 30))
+
+ screenshot_labeled_path = Path(self.screenshot_after_path).joinpath(f"{step}_labeled.png")
+ labeled_img = draw_bbox_multi(screenshot_path, screenshot_labeled_path, elem_list)
+
+ cv2.namedWindow("image", cv2.WINDOW_NORMAL)
+ cv2.imshow("image", labeled_img)
+ cv2.waitKey(0)
+ cv2.destroyAllWindows()
+
+ user_input = "xxx"
+ logger.info(
+ "Choose one of the following actions you want to perform on the current screen:\n"
+ "tap, text, long_press, swipe, stop"
+ )
+
+ while (
+ user_input.lower() != ActionOp.TAP.value
+ and user_input.lower() != ActionOp.TEXT.value
+ and user_input.lower() != ActionOp.LONG_PRESS.value
+ and user_input.lower() != ActionOp.SWIPE.value
+ and user_input.lower() != ActionOp.STOP.value
+ ):
+ user_input = input("user_input: ")
+
+ if user_input.lower() == ActionOp.TAP.value:
+ logger.info(f"Which element do you want to tap? Choose a numeric tag from 1 to {len(elem_list)}:")
+ user_input = "xxx"
+ while not user_input.isnumeric() or int(user_input) > len(elem_list) or int(user_input) < 1:
+ user_input = input("user_input: ")
+ tl, br = elem_list[int(user_input) - 1].bbox
+ x, y = (tl[0] + br[0]) // 2, (tl[1] + br[1]) // 2
+ action = EnvAction(action_type=EnvActionType.SYSTEM_TAP, coord=(x, y))
+ log_str = f"tap({int(user_input)}):::{elem_list[int(user_input) - 1].uid}\n"
+ elif user_input.lower() == ActionOp.TEXT.value:
+ logger.info(
+ f"Which element do you want to input the text string? Choose a numeric tag from 1 to "
+ f"{len(elem_list)}:"
+ )
+ input_area = "xxx"
+ while not input_area.isnumeric() or int(input_area) > len(elem_list) or int(input_area) < 1:
+ input_area = input("user_input: ")
+ logger.info("Enter your input text below:")
+ user_input = ""
+ while not user_input:
+ user_input = input("user_input: ")
+ action = EnvAction(action_type=EnvActionType.USER_INPUT, input_txt=user_input)
+ log_str = f"text({input_area}:sep:'{user_input}'):::{elem_list[int(input_area) - 1].uid}\n"
+ elif user_input.lower() == ActionOp.LONG_PRESS.value:
+ logger.info(
+ f"Which element do you want to long press? Choose a numeric tag from 1 to {len(elem_list)}:"
+ )
+ user_input = "xxx"
+ while not user_input.isnumeric() or int(user_input) > len(elem_list) or int(user_input) < 1:
+ user_input = input("user_input: ")
+ tl, br = elem_list[int(user_input) - 1].bbox
+ x, y = (tl[0] + br[0]) // 2, (tl[1] + br[1]) // 2
+ action = EnvAction(action_type=EnvActionType.USER_LONGPRESS, coord=(x, y))
+ log_str = f"long_press({int(user_input)}):::{elem_list[int(user_input) - 1].uid}\n"
+ elif user_input.lower() == ActionOp.SWIPE.value:
+ logger.info(
+ "What is the direction of your swipe? Choose one from the following options:\n"
+ "up, down, left, right"
+ )
+ user_input = ""
+ while (
+ user_input != SwipeOp.UP.value
+ and user_input != SwipeOp.DOWN.value
+ and user_input != SwipeOp.LEFT.value
+ and user_input != SwipeOp.RIGHT.value
+ ):
+ user_input = input("user_input: ")
+ swipe_dir = user_input
+ logger.info(f"Which element do you want to swipe? Choose a numeric tag from 1 to {len(elem_list)}:")
+ while not user_input.isnumeric() or int(user_input) > len(elem_list) or int(user_input) < 1:
+ user_input = input("user_input: ")
+ tl, br = elem_list[int(user_input) - 1].bbox
+ x, y = (tl[0] + br[0]) // 2, (tl[1] + br[1]) // 2
+
+ action = EnvAction(action_type=EnvActionType.USER_SWIPE, coord=(x, y), orient=swipe_dir)
+ log_str = f"swipe({int(user_input)}:sep:{swipe_dir}):::{elem_list[int(user_input) - 1].uid}\n"
+ elif user_input.lower() == ActionOp.STOP.value:
+ record_file.write("stop\n")
+ record_file.close()
+ break
+ else:
+ break
+
+ obs, _, _, _, info = env.step(action)
+ action_res = info["res"]
+ if action_res == ADB_EXEC_FAIL:
+ return AndroidActionOutput(action_state=RunState.FAIL)
+ record_file.write(log_str)
+
+ time.sleep(1)
+
+ return AndroidActionOutput(action_state=RunState.SUCCESS)
diff --git a/metagpt/ext/android_assistant/actions/parse_record.py b/metagpt/ext/android_assistant/actions/parse_record.py
new file mode 100644
index 000000000..304daf655
--- /dev/null
+++ b/metagpt/ext/android_assistant/actions/parse_record.py
@@ -0,0 +1,137 @@
+#!/usr/bin/env python
+# -*- coding: utf-8 -*-
+# @Desc : parse record to generate learned standard operations in stage=learn & mode=manual,
+# LIKE scripts/document_generation.py
+
+import ast
+import re
+from pathlib import Path
+
+from metagpt.actions.action import Action
+from metagpt.config2 import config
+from metagpt.ext.android_assistant.actions.parse_record_an import RECORD_PARSE_NODE
+from metagpt.ext.android_assistant.prompts.operation_prompt import (
+ long_press_doc_template,
+ refine_doc_suffix,
+ swipe_doc_template,
+ tap_doc_template,
+ text_doc_template,
+)
+from metagpt.ext.android_assistant.utils.schema import (
+ ActionOp,
+ AndroidActionOutput,
+ RecordLogItem,
+ RunState,
+ SwipeOp,
+)
+from metagpt.logs import logger
+from metagpt.utils.common import encode_image
+
+
+class ParseRecord(Action):
+ name: str = "ParseRecord"
+ record_path: Path = ""
+ task_desc_path: Path = ""
+ screenshot_before_path: Path = ""
+ screenshot_after_path: Path = ""
+
+ async def run(self, task_dir: Path, docs_dir: Path):
+ doc_count = 0
+ self.record_path = Path(task_dir) / "record.txt"
+ self.task_desc_path = Path(task_dir) / "task_desc.txt"
+ self.screenshot_before_path = Path(task_dir) / "raw_screenshots"
+ self.screenshot_after_path = Path(task_dir) / "labeled_screenshots"
+ for path in [self.screenshot_before_path, self.screenshot_after_path]:
+ path.mkdir(parents=True, exist_ok=True)
+
+ task_desc = self.task_desc_path.read_text()
+ extra_config = config.extra
+
+ with open(self.record_path, "r") as record_file:
+ record_step_count = len(record_file.readlines()) - 1
+ record_file.seek(0)
+ for step in range(1, record_step_count + 1):
+ img_before_base64 = encode_image(self.screenshot_after_path.joinpath(f"{step}_labeled.png"))
+ img_after_base64 = encode_image(self.screenshot_after_path.joinpath(f"{step + 1}_labeled.png"))
+ rec = record_file.readline().strip()
+ action, resource_id = rec.split(":::")
+ action_type = action.split("(")[0]
+ # 构建Prompt
+ action_param = re.findall(r"\((.*?)\)", action)[0]
+ if action_type == ActionOp.TAP.value:
+ prompt_template = tap_doc_template
+ context = prompt_template.format(ui_element=action_param)
+ elif action_type == ActionOp.TEXT.value:
+ input_area, input_text = action_param.split(":sep:")
+ prompt_template = text_doc_template
+ context = prompt_template.format(ui_element=input_area)
+ elif action_type == ActionOp.LONG_PRESS.value:
+ prompt_template = long_press_doc_template
+ context = prompt_template.format(ui_element=action_param)
+ elif action_type == ActionOp.SWIPE.value:
+ swipe_area, swipe_dir = action_param.split(":sep:")
+ if swipe_dir == SwipeOp.UP.value or swipe_dir == SwipeOp.DOWN.value:
+ action_type = ActionOp.VERTICAL_SWIPE.value
+ elif swipe_dir == SwipeOp.LEFT.value or swipe_dir == SwipeOp.RIGHT.value:
+ action_type = ActionOp.HORIZONTAL_SWIPE.value
+ prompt_template = swipe_doc_template
+ context = prompt_template.format(swipe_dir=swipe_dir, ui_element=swipe_area)
+ else:
+ break
+ context = context.format(task_desc=task_desc)
+
+ doc_name = resource_id + ".txt"
+ doc_path = docs_dir.joinpath(doc_name)
+
+ if doc_path.exists():
+ try:
+ doc_content = ast.literal_eval(doc_path.read_text())
+ except Exception as exp:
+ logger.error(f"ast parse doc: {doc_path} failed, exp: {exp}")
+ continue
+
+ if doc_content[action_type]:
+ if extra_config.get("doc_refine", False):
+ refine_context = refine_doc_suffix.format(old_doc=doc_content[action_type])
+ context += refine_context
+ logger.info(
+ f"Documentation for the element {resource_id} already exists. The doc will be "
+ f"refined based on the latest demo."
+ )
+ else:
+ logger.info(
+ f"Documentation for the element {resource_id} already exists. Turn on DOC_REFINE "
+ f"in the config file if needed."
+ )
+ continue
+ else:
+ doc_content = {"tap": "", "text": "", "v_swipe": "", "h_swipe": "", "long_press": ""}
+
+ logger.info(f"Waiting for GPT-4V to generate documentation for the element {resource_id}")
+ node = await RECORD_PARSE_NODE.fill(
+ context=context, llm=self.llm, images=[img_before_base64, img_after_base64]
+ )
+ if "error" in node.content:
+ return AndroidActionOutput(action_state=RunState.FAIL)
+ log_path = task_dir.joinpath("log_parse_record.txt")
+ prompt = node.compile(context=context, schema="json", mode="auto")
+ msg = node.content
+ doc_content[action_type] = msg
+
+ with open(log_path, "a") as logfile:
+ log_item = RecordLogItem(
+ step=step,
+ prompt=prompt,
+ image_before=img_before_base64,
+ image_after=img_after_base64,
+ response=node.content,
+ )
+ logfile.write(log_item.model_dump_json() + "\n")
+ with open(doc_path, "w") as outfile:
+ outfile.write(str(doc_content))
+ doc_count += 1
+ logger.info(f"Documentation generated and saved to {doc_path}")
+
+ logger.info(f"Documentation generation phase completed. {doc_count} docs generated.")
+
+ return AndroidActionOutput(action_state=RunState.FINISH)
diff --git a/metagpt/ext/android_assistant/actions/parse_record_an.py b/metagpt/ext/android_assistant/actions/parse_record_an.py
new file mode 100644
index 000000000..210c93e23
--- /dev/null
+++ b/metagpt/ext/android_assistant/actions/parse_record_an.py
@@ -0,0 +1,32 @@
+#!/usr/bin/env python
+# -*- coding: utf-8 -*-
+# @Desc : the ActionNode to parse record
+
+from metagpt.actions.action_node import ActionNode
+
+OBSERVATION = ActionNode(
+ key="Observation",
+ expected_type=str,
+ instruction="Provide a description of your observations of the two images. "
+ "Subsequently, delineate the distinctions between the first image and the second one.",
+ example="",
+)
+
+THOUGHT = ActionNode(
+ key="Thought",
+ expected_type=str,
+ instruction="Consider the impact of Action acting on UI elements.",
+ example="",
+)
+
+DESCRIPTION = ActionNode(
+ key="Description",
+ expected_type=str,
+ instruction="Describe the functionality of the UI element concisely in one or two sentences Do not include "
+ "the numeric tag in your description",
+ example="",
+)
+
+NODES = [OBSERVATION, THOUGHT, DESCRIPTION]
+
+RECORD_PARSE_NODE = ActionNode.from_children("RecordParse", NODES)
diff --git a/metagpt/ext/android_assistant/actions/screenshot_parse.py b/metagpt/ext/android_assistant/actions/screenshot_parse.py
new file mode 100644
index 000000000..4d8bb0e1e
--- /dev/null
+++ b/metagpt/ext/android_assistant/actions/screenshot_parse.py
@@ -0,0 +1,204 @@
+#!/usr/bin/env python
+# -*- coding: utf-8 -*-
+# @Desc : LIKE scripts/task_executor.py in stage=act
+
+import ast
+from pathlib import Path
+
+from metagpt.actions.action import Action
+from metagpt.config2 import config
+from metagpt.environment.android.android_env import AndroidEnv
+from metagpt.environment.android.const import ADB_EXEC_FAIL
+from metagpt.environment.android.env_space import (
+ EnvAction,
+ EnvActionType,
+ EnvObsParams,
+ EnvObsType,
+)
+from metagpt.ext.android_assistant.actions.screenshot_parse_an import (
+ SCREENSHOT_PARSE_NODE,
+)
+from metagpt.ext.android_assistant.prompts.assistant_prompt import (
+ screenshot_parse_template,
+ screenshot_parse_with_grid_template,
+)
+from metagpt.ext.android_assistant.utils.schema import (
+ AndroidActionOutput,
+ AndroidElement,
+ GridOpParam,
+ LongPressGridOpParam,
+ LongPressOpParam,
+ OpLogItem,
+ RunState,
+ SwipeGridOpParam,
+ SwipeOpParam,
+ TapGridOpParam,
+ TapOpParam,
+ TextOpParam,
+)
+from metagpt.ext.android_assistant.utils.utils import (
+ area_to_xy,
+ draw_bbox_multi,
+ draw_grid,
+ elem_bbox_to_xy,
+ screenshot_parse_extract,
+ traverse_xml_tree,
+)
+from metagpt.logs import logger
+from metagpt.utils.common import encode_image
+
+
+class ScreenshotParse(Action):
+ name: str = "ScreenshotParse"
+
+ def _makeup_ui_document(self, elem_list: list[AndroidElement], docs_idr: Path, use_exist_doc: bool = True) -> str:
+ if not use_exist_doc:
+ return ""
+
+ ui_doc = """
+You also have access to the following documentations that describes the functionalities of UI
+elements you can interact on the screen. These docs are crucial for you to determine the target of your
+next action. You should always prioritize these documented elements for interaction: """
+ for i, elem in enumerate(elem_list):
+ doc_path = docs_idr.joinpath(f"{elem.uid}.txt")
+ if not doc_path.exists():
+ continue
+ try:
+ doc_content = ast.literal_eval(doc_path.read_text())
+ except Exception as exp:
+ logger.error(f"ast parse doc: {doc_path} failed, exp: {exp}")
+ continue
+
+ ui_doc += f"Documentation of UI element labeled with the numeric tag '{i + 1}':\n"
+ if doc_content["tap"]:
+ ui_doc += f"This UI element is clickable. {doc_content['tap']}\n\n"
+ if doc_content["text"]:
+ ui_doc += (
+ f"This UI element can receive text input. The text input is used for the following "
+ f"purposes: {doc_content['text']}\n\n"
+ )
+ if doc_content["long_press"]:
+ ui_doc += f"This UI element is long clickable. {doc_content['long_press']}\n\n"
+ if doc_content["v_swipe"]:
+ ui_doc += (
+ f"This element can be swiped directly without tapping. You can swipe vertically on "
+ f"this UI element. {doc_content['v_swipe']}\n\n"
+ )
+ if doc_content["h_swipe"]:
+ ui_doc += (
+ f"This element can be swiped directly without tapping. You can swipe horizontally on "
+ f"this UI element. {doc_content['h_swipe']}\n\n"
+ )
+ return ui_doc
+
+ async def run(
+ self,
+ round_count: int,
+ task_desc: str,
+ last_act: str,
+ task_dir: Path,
+ docs_dir: Path,
+ grid_on: bool,
+ env: AndroidEnv,
+ ):
+ extra_config = config.extra
+ for path in [task_dir, docs_dir]:
+ path.mkdir(parents=True, exist_ok=True)
+ screenshot_path: Path = env.observe(
+ EnvObsParams(obs_type=EnvObsType.GET_SCREENSHOT, ss_name=f"{round_count}_before", local_save_dir=task_dir)
+ )
+ xml_path: Path = env.observe(
+ EnvObsParams(obs_type=EnvObsType.GET_XML, xml_name=f"{round_count}", local_save_dir=task_dir)
+ )
+ if not screenshot_path.exists() or not xml_path.exists():
+ return AndroidActionOutput(action_state=RunState.FAIL)
+
+ clickable_list = []
+ focusable_list = []
+ traverse_xml_tree(xml_path, clickable_list, "clickable", True)
+ traverse_xml_tree(xml_path, focusable_list, "focusable", True)
+ elem_list: list[AndroidElement] = clickable_list.copy()
+ for elem in focusable_list:
+ bbox = elem.bbox
+ center = (bbox[0][0] + bbox[1][0]) // 2, (bbox[0][1] + bbox[1][1]) // 2
+ close = False
+ for e in clickable_list:
+ bbox = e.bbox
+ center_ = (bbox[0][0] + bbox[1][0]) // 2, (bbox[0][1] + bbox[1][1]) // 2
+ dist = (abs(center[0] - center_[0]) ** 2 + abs(center[1] - center_[1]) ** 2) ** 0.5
+ if dist <= extra_config.get("min_dist", 30):
+ close = True
+ break
+ if not close:
+ elem_list.append(elem)
+
+ screenshot_labeled_path = task_dir.joinpath(f"{round_count}_labeled.png")
+ draw_bbox_multi(screenshot_path, screenshot_labeled_path, elem_list)
+ img_base64 = encode_image(screenshot_labeled_path)
+
+ parse_template = screenshot_parse_with_grid_template if grid_on else screenshot_parse_template
+
+ if grid_on:
+ env.rows, env.cols = draw_grid(screenshot_path, task_dir / f"{round_count}_grid.png")
+
+ ui_doc = self._makeup_ui_document(elem_list, docs_dir)
+ context = parse_template.format(ui_document=ui_doc, task_description=task_desc, last_act=last_act)
+ node = await SCREENSHOT_PARSE_NODE.fill(context=context, llm=self.llm, images=[img_base64])
+
+ if "error" in node.content:
+ return AndroidActionOutput(action_state=RunState.FAIL)
+
+ prompt = node.compile(context=context, schema="json", mode="auto")
+ OpLogItem(step=round_count, prompt=prompt, image=str(screenshot_labeled_path), response=node.content)
+
+ op_param = screenshot_parse_extract(node.instruct_content.model_dump(), grid_on)
+ if op_param.param_state == RunState.FINISH:
+ logger.info(f"op_param: {op_param}")
+ return AndroidActionOutput(action_state=RunState.FINISH)
+ if op_param.param_state == RunState.FAIL:
+ return AndroidActionOutput(action_state=RunState.FAIL)
+
+ last_act = op_param.last_act
+ if isinstance(op_param, TapOpParam):
+ x, y = elem_bbox_to_xy(elem_list[op_param.area - 1].bbox)
+ action = EnvAction(action_type=EnvActionType.SYSTEM_TAP, coord=(x, y))
+ elif isinstance(op_param, TextOpParam):
+ action = EnvAction(action_type=EnvActionType.USER_INPUT, input_txt=op_param.input_str)
+ elif isinstance(op_param, LongPressOpParam):
+ x, y = elem_bbox_to_xy(elem_list[op_param.area - 1].bbox)
+ action = EnvAction(action_type=EnvActionType.USER_LONGPRESS, coord=(x, y))
+ elif isinstance(op_param, SwipeOpParam):
+ x, y = elem_bbox_to_xy(elem_list[op_param.area - 1].bbox)
+ action = EnvAction(
+ action_type=EnvActionType.USER_SWIPE, coord=(x, y), orient=op_param.swipe_orient, dist=op_param.dist
+ )
+ elif isinstance(op_param, GridOpParam):
+ grid_on = True
+ elif isinstance(op_param, TapGridOpParam) or isinstance(op_param, LongPressGridOpParam):
+ x, y = area_to_xy(op_param.area, op_param.subarea, env.width, env.height, env.rows, env.cols)
+ if isinstance(op_param, TapGridOpParam):
+ action = EnvAction(action_type=EnvActionType.SYSTEM_TAP, coord=(x, y))
+ else:
+ # LongPressGridOpParam
+ action = EnvAction(action_type=EnvActionType.USER_LONGPRESS, coord=(x, y))
+ elif isinstance(op_param, SwipeGridOpParam):
+ start_x, start_y = area_to_xy(
+ op_param.start_area, op_param.start_subarea, env.width, env.height, env.rows, env.cols
+ )
+ end_x, end_y = area_to_xy(
+ op_param.end_area, op_param.end_subarea, env.width, env.height, env.rows, env.cols
+ )
+ action = EnvAction(
+ action_type=EnvActionType.USER_SWIPE_TO, coord=(start_x, start_y), tgt_coord=(end_x, end_y)
+ )
+
+ if not grid_on:
+ obs, _, _, _, info = env.step(action)
+ action_res = info["res"]
+ if action_res == ADB_EXEC_FAIL:
+ return AndroidActionOutput(action_state=RunState.FAIL)
+
+ if op_param.act_name != "grid":
+ grid_on = False
+
+ return AndroidActionOutput(data={"grid_on": grid_on, "last_act": last_act})
diff --git a/metagpt/ext/android_assistant/actions/screenshot_parse_an.py b/metagpt/ext/android_assistant/actions/screenshot_parse_an.py
new file mode 100644
index 000000000..eb23ba934
--- /dev/null
+++ b/metagpt/ext/android_assistant/actions/screenshot_parse_an.py
@@ -0,0 +1,48 @@
+#!/usr/bin/env python
+# -*- coding: utf-8 -*-
+# @Desc : the ActionNode to parse screenshot
+
+from metagpt.actions.action_node import ActionNode
+
+OBSERVATION = ActionNode(
+ key="Observation", expected_type=str, instruction="Describe what you observe in the image", example=""
+)
+
+THOUGHT = ActionNode(
+ key="Thought",
+ expected_type=str,
+ instruction="To complete the given task, what is the next step I should do",
+ example="",
+)
+
+ACTION = ActionNode(
+ key="Action",
+ expected_type=str,
+ instruction="The function call with the correct parameters to proceed with the task. If you believe the task is "
+ "completed or there is nothing to be done, you should output FINISH. You cannot output anything else "
+ "except a function call or FINISH in this field.",
+ example="",
+)
+
+SUMMARY = ActionNode(
+ key="Summary",
+ expected_type=str,
+ instruction="Summarize your past actions along with your latest action in one or two sentences. Do not include "
+ "the numeric tag in your summary",
+ example="",
+)
+
+SUMMARY_GRID = ActionNode(
+ key="Summary",
+ expected_type=str,
+ instruction="Summarize your past actions along with your latest action in one or two sentences. Do not include "
+ "the grid area number in your summary",
+ example="",
+)
+
+NODES = [OBSERVATION, THOUGHT, ACTION, SUMMARY]
+
+NODES_GRID = [OBSERVATION, THOUGHT, ACTION, SUMMARY_GRID]
+
+SCREENSHOT_PARSE_NODE = ActionNode.from_children("ScreenshotParse", NODES)
+SCREENSHOT_PARSE_GRID_NODE = ActionNode.from_children("ScreenshotParseGrid", NODES_GRID)
diff --git a/metagpt/ext/android_assistant/actions/self_learn_and_reflect.py b/metagpt/ext/android_assistant/actions/self_learn_and_reflect.py
new file mode 100644
index 000000000..5e9cfbb45
--- /dev/null
+++ b/metagpt/ext/android_assistant/actions/self_learn_and_reflect.py
@@ -0,0 +1,231 @@
+# !/usr/bin/env python
+# -*- coding: utf-8 -*-
+# @Desc : LIKE scripts/self_explorer.py in stage=learn & mode=auto self_explore_task stage
+
+import ast
+from pathlib import Path
+
+from metagpt.actions.action import Action
+from metagpt.config2 import config
+from metagpt.environment.android.android_env import AndroidEnv
+from metagpt.environment.android.const import ADB_EXEC_FAIL
+from metagpt.environment.android.env_space import (
+ EnvAction,
+ EnvActionType,
+ EnvObsParams,
+ EnvObsType,
+)
+from metagpt.ext.android_assistant.actions.screenshot_parse_an import (
+ SCREENSHOT_PARSE_NODE,
+)
+from metagpt.ext.android_assistant.actions.self_learn_reflect_an import (
+ SELF_LEARN_REFLECT_NODE,
+)
+from metagpt.ext.android_assistant.prompts.assistant_prompt import (
+ screenshot_parse_self_explore_reflect_template as reflect_template,
+)
+from metagpt.ext.android_assistant.prompts.assistant_prompt import (
+ screenshot_parse_self_explore_template,
+)
+from metagpt.ext.android_assistant.utils.schema import (
+ ActionOp,
+ AndroidActionOutput,
+ AndroidElement,
+ Decision,
+ DocContent,
+ LongPressOpParam,
+ OpLogItem,
+ ReflectLogItem,
+ RunState,
+ SwipeOp,
+ SwipeOpParam,
+ TapOpParam,
+ TextOpParam,
+)
+from metagpt.ext.android_assistant.utils.utils import (
+ draw_bbox_multi,
+ elem_bbox_to_xy,
+ elem_list_from_xml_tree,
+ reflect_parse_extarct,
+ screenshot_parse_extract,
+)
+from metagpt.logs import logger
+from metagpt.utils.common import encode_image
+
+
+class SelfLearnAndReflect(Action):
+ name: str = "SelfLearnAndReflect"
+
+ useless_list: list[str] = [] # store useless elements uid
+
+ screenshot_before_path: str = ""
+ screenshot_before_base64: str = ""
+ elem_list: list[AndroidElement] = []
+ swipe_orient: str = "up"
+ act_name: str = ""
+ ui_area: int = -1
+
+ async def run(
+ self, round_count: int, task_desc: str, last_act: str, task_dir: Path, docs_dir: Path, env: AndroidEnv
+ ) -> AndroidActionOutput:
+ for path in [task_dir, docs_dir]:
+ path.mkdir(parents=True, exist_ok=True)
+ resp = await self.run_self_learn(round_count, task_desc, last_act, task_dir, env)
+ if resp.action_state != RunState.SUCCESS:
+ return resp
+
+ resp = await self.run_reflect(round_count, task_desc, last_act, task_dir, docs_dir, env)
+ return resp
+
+ async def run_self_learn(
+ self, round_count: int, task_desc: str, last_act: str, task_dir: Path, env: AndroidEnv
+ ) -> AndroidActionOutput:
+ extra_config = config.extra
+ screenshot_path: Path = env.observe(
+ EnvObsParams(obs_type=EnvObsType.GET_SCREENSHOT, ss_name=f"{round_count}_before", local_save_dir=task_dir)
+ )
+ xml_path: Path = env.observe(
+ EnvObsParams(obs_type=EnvObsType.GET_XML, xml_name=f"{round_count}", local_save_dir=task_dir)
+ )
+ if not screenshot_path.exists() or not xml_path.exists():
+ return AndroidActionOutput(action_state=RunState.FAIL)
+
+ elem_list = elem_list_from_xml_tree(xml_path, self.useless_list, extra_config.get("min_dist", 30))
+
+ screenshot_before_labeled_path = task_dir.joinpath(f"{round_count}_before_labeled.png")
+ draw_bbox_multi(screenshot_path, screenshot_before_labeled_path, elem_list)
+ img_base64 = encode_image(screenshot_before_labeled_path)
+ self.screenshot_before_base64 = img_base64
+ self.screenshot_before_path = screenshot_before_labeled_path
+
+ self_explore_template = screenshot_parse_self_explore_template
+ context = self_explore_template.format(task_description=task_desc, last_act=last_act)
+
+ node = await SCREENSHOT_PARSE_NODE.fill(context=context, llm=self.llm, images=[img_base64])
+ logger.debug(f"fill result:{node}")
+ if "error" in node.content:
+ return AndroidActionOutput(action_state=RunState.FAIL)
+ prompt = node.compile(context=context, schema="json", mode="auto")
+ # Modify WindowsPath to Str
+ OpLogItem(step=round_count, prompt=prompt, image=str(screenshot_before_labeled_path), response=node.content)
+ op_param = screenshot_parse_extract(node.instruct_content.model_dump(), grid_on=False)
+ # TODO Modify Op_param. When op_param.action is FINISH, how to solve this ?
+ if op_param.param_state == RunState.FINISH:
+ return AndroidActionOutput(action_state=RunState.FINISH)
+ if op_param.param_state == RunState.FAIL:
+ return AndroidActionOutput(action_state=RunState.FAIL)
+
+ if isinstance(op_param, TapOpParam):
+ self.ui_area = op_param.area
+ x, y = elem_bbox_to_xy(elem_list[op_param.area - 1].bbox)
+ action = EnvAction(action_type=EnvActionType.SYSTEM_TAP, coord=(x, y))
+ elif isinstance(op_param, TextOpParam):
+ action = EnvAction(action_type=EnvActionType.USER_INPUT, input_txt=op_param.input_str)
+ elif isinstance(op_param, LongPressOpParam):
+ self.ui_area = op_param.area
+ x, y = elem_bbox_to_xy(elem_list[op_param.area - 1].bbox)
+ action = EnvAction(action_type=EnvActionType.USER_LONGPRESS, coord=(x, y))
+ elif isinstance(op_param, SwipeOpParam):
+ self.ui_area = op_param.area
+ self.swipe_orient = op_param.swipe_orient
+ x, y = elem_bbox_to_xy(elem_list[op_param.area - 1].bbox)
+ action = EnvAction(
+ action_type=EnvActionType.USER_SWIPE, coord=(x, y), orient=op_param.swipe_orient, dist=op_param.dist
+ )
+
+ obs, _, _, _, info = env.step(action)
+ action_res = info["res"]
+ if action_res == ADB_EXEC_FAIL:
+ return AndroidActionOutput(action_state=RunState.FAIL)
+
+ self.elem_list = elem_list
+ self.act_name = op_param.act_name
+ return AndroidActionOutput()
+
+ async def run_reflect(
+ self, round_count: int, task_desc: str, last_act: str, task_dir: Path, docs_dir: Path, env: AndroidEnv
+ ) -> AndroidActionOutput:
+ screenshot_path: Path = env.observe(
+ EnvObsParams(obs_type=EnvObsType.GET_SCREENSHOT, ss_name=f"{round_count}_after", local_save_dir=task_dir)
+ )
+ if not screenshot_path.exists():
+ return AndroidActionOutput(action_state=RunState.FAIL)
+
+ screenshot_after_labeled_path = task_dir.joinpath(f"{round_count}_after_labeled.png")
+ draw_bbox_multi(screenshot_path, screenshot_after_labeled_path, elem_list=self.elem_list)
+ img_base64 = encode_image(screenshot_after_labeled_path)
+ if self.act_name == ActionOp.TAP.value:
+ action = "tapping"
+ elif self.act_name == ActionOp.LONG_PRESS.value:
+ action = "long pressing"
+ elif self.act_name == ActionOp.SWIPE.value:
+ action = "swiping"
+ if self.swipe_orient == SwipeOp.UP.value or self.swipe_orient == SwipeOp.DOWN.value:
+ action = "v_swipe"
+ elif self.swipe_orient == SwipeOp.LEFT.value or self.swipe_orient == SwipeOp.RIGHT.value:
+ action = "h_swipe"
+ else:
+ # TODO Test for assignment, This error is eupiped with the next.
+ logger.warning(f"Current action name parse failed, it's `{self.act_name}`")
+ action = None
+ context = reflect_template.format(
+ action=action, ui_element=str(self.ui_area), task_desc=task_desc, last_act=last_act
+ )
+ node = await SELF_LEARN_REFLECT_NODE.fill(
+ context=context, llm=self.llm, images=[self.screenshot_before_base64, img_base64]
+ )
+
+ if "error" in node.content:
+ return AndroidActionOutput(action_state=RunState.FAIL)
+
+ prompt = node.compile(context=context, schema="json", mode="auto")
+ ReflectLogItem(
+ step=round_count,
+ prompt=prompt,
+ image_before=str(self.screenshot_before_path),
+ image_after=str(screenshot_after_labeled_path),
+ response=node.content,
+ )
+
+ op_param = reflect_parse_extarct(node.instruct_content.model_dump())
+ if op_param.param_state == RunState.FINISH:
+ return AndroidActionOutput(action_state=RunState.FINISH)
+ if op_param.param_state == RunState.FAIL:
+ return AndroidActionOutput(action_state=RunState.FAIL)
+
+ logger.info(
+ f"reflect_parse_extarct decision: {op_param.decision}, "
+ f"elem_list size: {len(self.elem_list)}, ui_area: {self.ui_area}"
+ )
+ # TODO here will cause `IndexError: list index out of range`.
+ # Maybe you should clink back to the desktop in the simulator
+ resource_id = self.elem_list[int(self.ui_area) - 1].uid
+ if op_param.decision == Decision.INEFFECTIVE.value:
+ self.useless_list.append(resource_id)
+ last_act = "NONE" # TODO global
+ elif op_param.decision in [Decision.BACK.value, Decision.CONTINUE.value, Decision.SUCCESS.value]:
+ if op_param.decision in [Decision.BACK.value, Decision.CONTINUE.value]:
+ self.useless_list.append(resource_id)
+ last_act = "NONE"
+ if op_param.decision == Decision.BACK.value:
+ action = EnvAction(action_type=EnvActionType.SYSTEM_BACK)
+ obs, _, _, _, info = env.step(action)
+ if info["res"] == ADB_EXEC_FAIL:
+ return AndroidActionOutput(action_state=RunState.FAIL)
+ doc = op_param.documentation
+ doc_path = docs_dir.joinpath(f"{resource_id}.txt")
+ if doc_path.exists():
+ try:
+ doc_content = ast.literal_eval(doc_path.read_text())
+ except Exception as exp:
+ logger.error(f"ast parse doc: {doc_path} failed, exp: {exp}")
+ return AndroidActionOutput(action_state=RunState.FAIL)
+
+ if doc_content[self.act_name]:
+ logger.info(f"Documentation for the element {resource_id} already exists.")
+ return AndroidActionOutput(action_state=RunState.FAIL)
+ else:
+ doc_content = DocContent()
+ setattr(doc_content, self.act_name, doc)
+ doc_path.write_text(str(doc_content))
+ return AndroidActionOutput(data={"last_act": last_act})
diff --git a/metagpt/ext/android_assistant/actions/self_learn_reflect_an.py b/metagpt/ext/android_assistant/actions/self_learn_reflect_an.py
new file mode 100644
index 000000000..305b7376a
--- /dev/null
+++ b/metagpt/ext/android_assistant/actions/self_learn_reflect_an.py
@@ -0,0 +1,21 @@
+#!/usr/bin/env python
+# -*- coding: utf-8 -*-
+# @Desc : the ActionNode to parse Reflection
+
+from metagpt.actions.action_node import ActionNode
+
+DECISION = ActionNode(
+ key="Decision", expected_type=str, instruction="explain why you made this decision", example="BACK"
+)
+
+
+THOUGHT = ActionNode(key="Thought", expected_type=str, instruction="explain why you made this decision", example="")
+
+
+DOCUMENTATION = ActionNode(
+ key="Documentation", expected_type=str, instruction="describe the function of the UI element", example=""
+)
+
+
+NODES = [DECISION, THOUGHT, DOCUMENTATION]
+SELF_LEARN_REFLECT_NODE = ActionNode.from_children("SelfLearnReflect", NODES)
diff --git a/metagpt/ext/android_assistant/prompts/__init__.py b/metagpt/ext/android_assistant/prompts/__init__.py
new file mode 100644
index 000000000..2bcf8efd0
--- /dev/null
+++ b/metagpt/ext/android_assistant/prompts/__init__.py
@@ -0,0 +1,3 @@
+#!/usr/bin/env python
+# -*- coding: utf-8 -*-
+# @Desc :
diff --git a/metagpt/ext/android_assistant/prompts/assistant_prompt.py b/metagpt/ext/android_assistant/prompts/assistant_prompt.py
new file mode 100644
index 000000000..34baf5841
--- /dev/null
+++ b/metagpt/ext/android_assistant/prompts/assistant_prompt.py
@@ -0,0 +1,168 @@
+#!/usr/bin/env python
+# -*- coding: utf-8 -*-
+# @Desc : the prompt templates of assistant learning and acting
+
+screenshot_parse_template = """You are an agent that is trained to perform some basic tasks on a smartphone. You will be given a
+smartphone screenshot. The interactive UI elements on the screenshot are labeled with numeric tags starting from 1. The
+numeric tag of each interactive element is located in the center of the element.
+
+You can call the following functions to control the smartphone:
+
+1. tap(element: int)
+This function is used to tap an UI element shown on the smartphone screen.
+"element" is a numeric tag assigned to an UI element shown on the smartphone screen.
+A simple use case can be tap(5), which taps the UI element labeled with the number 5.
+
+2. text(text_input: str)
+This function is used to insert text input in an input field/box. text_input is the string you want to insert and must
+be wrapped with double quotation marks. A simple use case can be text("Hello, world!"), which inserts the string
+"Hello, world!" into the input area on the smartphone screen. This function is usually callable when you see a keyboard
+showing in the lower half of the screen.
+
+3. long_press(element: int)
+This function is used to long press an UI element shown on the smartphone screen.
+"element" is a numeric tag assigned to an UI element shown on the smartphone screen.
+A simple use case can be long_press(5), which long presses the UI element labeled with the number 5.
+
+4. swipe(element: int, direction: str, dist: str)
+This function is used to swipe an UI element shown on the smartphone screen, usually a scroll view or a slide bar.
+"element" is a numeric tag assigned to an UI element shown on the smartphone screen. "direction" is a string that
+represents one of the four directions: up, down, left, right. "direction" must be wrapped with double quotation
+marks. "dist" determines the distance of the swipe and can be one of the three options: short, medium, long. You should
+choose the appropriate distance option according to your need.
+A simple use case can be swipe(21, "up", "medium"), which swipes up the UI element labeled with the number 21 for a
+medium distance.
+
+5. grid()
+You should call this function when you find the element you want to interact with is not labeled with a numeric tag and
+other elements with numeric tags cannot help with the task. The function will bring up a grid overlay to divide the
+smartphone screen into small areas and this will give you more freedom to choose any part of the screen to tap, long
+press, or swipe.
+{ui_document}
+The task you need to complete is to: {task_description}. Your past actions to proceed with this task are summarized as
+follows: {last_act}
+Now, given the documentation and the following labeled screenshot, you need to think and call the function needed to
+proceed with the task. Your output should include three parts in the given format:
+
+You can only take one action at a time, so please directly call the function."""
+
+screenshot_parse_with_grid_template = """You are an agent that is trained to perform some basic tasks on a smartphone. You will be given
+a smartphone screenshot overlaid by a grid. The grid divides the screenshot into small square areas. Each area is
+labeled with an integer in the top-left corner.
+
+You can call the following functions to control the smartphone:
+
+1. tap(area: int, subarea: str)
+This function is used to tap a grid area shown on the smartphone screen. "area" is the integer label assigned to a grid
+area shown on the smartphone screen. "subarea" is a string representing the exact location to tap within the grid area.
+It can take one of the nine values: center, top-left, top, top-right, left, right, bottom-left, bottom, and
+bottom-right.
+A simple use case can be tap(5, "center"), which taps the exact center of the grid area labeled with the number 5.
+
+2. long_press(area: int, subarea: str)
+This function is used to long press a grid area shown on the smartphone screen. "area" is the integer label assigned to
+a grid area shown on the smartphone screen. "subarea" is a string representing the exact location to long press within
+the grid area. It can take one of the nine values: center, top-left, top, top-right, left, right, bottom-left, bottom,
+and bottom-right.
+A simple use case can be long_press(7, "top-left"), which long presses the top left part of the grid area labeled with
+the number 7.
+
+3. swipe(start_area: int, start_subarea: str, end_area: int, end_subarea: str)
+This function is used to perform a swipe action on the smartphone screen, especially when you want to interact with a
+scroll view or a slide bar. "start_area" is the integer label assigned to the grid area which marks the starting
+location of the swipe. "start_subarea" is a string representing the exact location to begin the swipe within the grid
+area. "end_area" is the integer label assigned to the grid area which marks the ending location of the swipe.
+"end_subarea" is a string representing the exact location to end the swipe within the grid area.
+The two subarea parameters can take one of the nine values: center, top-left, top, top-right, left, right, bottom-left,
+bottom, and bottom-right.
+A simple use case can be swipe(21, "center", 25, "right"), which performs a swipe starting from the center of grid area
+21 to the right part of grid area 25.
+
+The task you need to complete is to: {task_description}. Your past actions to proceed with this task are summarized as
+follows: {last_act}
+Now, given the following labeled screenshot, you need to think and call the function needed to proceed with the task.
+Your output should include three parts in the given format:
+
+You can only take one action at a time, so please directly call the function."""
+
+screenshot_parse_self_explore_template = """You are an agent that is trained to complete certain tasks on a smartphone. You will be
+given a screenshot of a smartphone app. The interactive UI elements on the screenshot are labeled with numeric tags
+starting from 1.
+
+You can call the following functions to interact with those labeled elements to control the smartphone:
+
+1. tap(element: int)
+This function is used to tap an UI element shown on the smartphone screen.
+"element" is a numeric tag assigned to an UI element shown on the smartphone screen.
+A simple use case can be tap(5), which taps the UI element labeled with the number 5.
+
+2. text(text_input: str)
+This function is used to insert text input in an input field/box. text_input is the string you want to insert and must
+be wrapped with double quotation marks. A simple use case can be text("Hello, world!"), which inserts the string
+"Hello, world!" into the input area on the smartphone screen. This function is only callable when you see a keyboard
+showing in the lower half of the screen.
+
+3. long_press(element: int)
+This function is used to long press an UI element shown on the smartphone screen.
+"element" is a numeric tag assigned to an UI element shown on the smartphone screen.
+A simple use case can be long_press(5), which long presses the UI element labeled with the number 5.
+
+4. swipe(element: int, direction: str, dist: str)
+This function is used to swipe an UI element shown on the smartphone screen, usually a scroll view or a slide bar.
+"element" is a numeric tag assigned to an UI element shown on the smartphone screen. "direction" is a string that
+represents one of the four directions: up, down, left, right. "direction" must be wrapped with double quotation
+marks. "dist" determines the distance of the swipe and can be one of the three options: short, medium, long. You should
+choose the appropriate distance option according to your need.
+A simple use case can be swipe(21, "up", "medium"), which swipes up the UI element labeled with the number 21 for a
+medium distance.
+
+The task you need to complete is to {task_description}. Your past actions to proceed with this task are summarized as
+follows: {last_act}
+Now, given the following labeled screenshot, you need to think and call the function needed to proceed with the task.
+Your output should include three parts in the given format:
+
+You can only take one action at a time, so please directly call the function."""
+
+screenshot_parse_self_explore_reflect_template = """I will give you screenshots of a mobile app before and after {action} the UI
+element labeled with the number '{ui_element}' on the first screenshot. The numeric tag of each element is located at
+the center of the element. The action of {action} this UI element was described as follows:
+{last_act}
+The action was also an attempt to proceed with a larger task, which is to {task_desc}. Your job is to carefully analyze
+the difference between the two screenshots to determine if the action is in accord with the description above and at
+the same time effectively moved the task forward. Your output should be determined based on the following situations:
+1. BACK
+If you think the action navigated you to a page where you cannot proceed with the given task, you should go back to the
+previous interface. At the same time, describe the functionality of the UI element concisely in one or two sentences by
+observing the difference between the two screenshots. Notice that your description of the UI element should focus on
+the general function. Never include the numeric tag of the UI element in your description. You can use pronouns such as
+"the UI element" to refer to the element. Your output should be in the following format:
+Decision: BACK
+Thought: is the role whose schedule we are editing right now.
+ scratch = role.rc.scratch
+ #