LLM-agents Project Release Note
Overview
The primary goal of our LLM-agents project is to leverage Language Models to devise strategies for addressing critical policy challenges in cancer care. With specific focus on European Union policy aims, we have drawn approximately ten policy objectives from an official EU document.
In the initial stages, we manually input these policy objectives into GPT-based models and assessed their generated strategies' quality. We especially noted that the GPT agents, namely BabyAGI and Camel, exhibited promising results in producing more innovative problem-solving strategies.
However, the manual approach was not scalable, hence the need for automation.
Achievements
1. Automated Strategy Generation and Evaluation
We've successfully automated the strategy generation process for the remaining policy objectives. This not only streamlines the evaluation but also allows us to manage multiple prompts seamlessly.
2. Web-Based Interface for Expert Evaluation
Developed a user-friendly web interface where experts can:
- Access and review generated strategic plans.
- Evaluate and rate strategies based on criteria like accuracy, relevance, creativity, specificity, feasibility, and the presence of red flags.
3. Database Integration
All the strategies, once generated, are parsed from markdown files and stored in an SQLite database, structured as 'Summaries'. Expert evaluations, on the other hand, are logged in the 'Scores' table. This structured approach ensures organized storage and easy retrieval of both generated strategies and their corresponding evaluations.
4. Future Goals
We're gearing up to:
- Further refine our evaluation metrics based on expert feedback.
- Expand the policy objectives for more extensive strategy generation.
- Continuously improve the quality and relevance of generated strategies.
Flowcharts
To understand the workflow, we've detailed our process in two flowcharts:
- [Web-Based User Interface for Expert Evaluation]
- [Database Creation and Data Integration]
Looking Ahead
As we continue our work on the LLM-agents project, we invite collaboration, suggestions, and feedback from the community. We believe that with collective efforts, we can enhance our models' capabilities and make significant strides in cancer care policy formulation.
For more information or to contribute, check out our project repository.