Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Update Readme #64

Open
wants to merge 4 commits into
base: master
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
247 changes: 139 additions & 108 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,125 +1,54 @@
# KAG: Knowledge Augmented Generation

[中文版文档](./README_cn.md)
[日本語版ドキュメント](./README_ja.md)
<div align="center">
<a href="https://spg.openkg.cn/en-US">
<img src="./_static/images/OpenSPG-1.png" width="520" alt="openspg logo">
</a>
</div>

## 1. What is KAG
<p align="center">
<a href="./README.md">English</a> |
<a href="./README_cn.md">简体中文</a> |
<a href="./README_ja.md">日本語版ドキュメント</a>
</p>

Retrieval Augmentation Generation (RAG) technology promotes the integration of domain applications with large language models. However, RAG has problems such as a large gap between vector similarity and knowledge reasoning correlation, and insensitivity to knowledge logic (such as numerical values, time relationships, expert rules, etc.), which hinder the implementation of professional knowledge services.
<p align="center">
<a href='https://arxiv.org/pdf/2409.13731'><img src='https://img.shields.io/badge/arXiv-2409.13731-b31b1b'></a>
<a href="https://github.com/OpenSPG/KAG/releases/latest">
<img src="https://img.shields.io/github/v/release/OpenSPG/KAG?color=blue&label=Latest%20Release" alt="Latest Release">
</a>
<a href="https://github.com/OpenSPG/KAG/blob/main/LICENSE">
<img height="21" src="https://img.shields.io/badge/License-Apache--2.0-ffffff?labelColor=d4eaf7&color=2e6cc4" alt="license">
</a>
</p>

On October 24, 2024, OpenSPG released v0.5, officially releasing the professional domain knowledge service framework of knowledge augmented generation (KAG). The goal of KAG is to build a knowledge-enhanced LLM service framework in professional domains, supporting logical reasoning, factual Q&A, etc. KAG fully integrates the logical and factual characteristics of the KGs. Meanwhile, it uses OpenIE to lower the threshold for knowledgeization of domain documents and alleviates the sparsity problem of the KG through hybrid reasoning. As far as we know, KAG is the only RAG framework that supports logical reasoning and multi-hop factual Q&A. Its core features include:

* Knowledge and Chunk Mutual Indexing structure to integrate more complete contextual text information
* Knowledge alignment using conceptual semantic reasoning to alleviate the noise problem caused by OpenIE
* Schema-constrained knowledge construction to support the representation and construction of domain expert knowledge
* Logical form-guided hybrid reasoning and retrieval to support logical reasoning and multi-hop reasoning Q&A
## 1. What is KAG?

KAG is significantly better than NaiveRAG, HippoRAG and other methods in multi-hop Q&A tasks. The F1 score on hotpotQA is relatively increased by 19.6%, and the F1 score on 2wiki is relatively increased by 33.5%. We have successfully applied KAG to Ant Group's professional knowledge Q&A tasks, such as e-government Q&A and e-health Q&A, and the professionalism has been significantly improved compared to the traditional RAG method.
KAG is a logical reasoning and Q&A framework based on the [OpenSPG](https://github.com/OpenSPG/openspg) engine and large language models, which is used to build logical reasoning and Q&A solutions for vertical domain knowledge bases. KAG can effectively overcome the ambiguity of traditional RAG vector similarity calculation and the noise problem of GraphRAG introduced by OpenIE. KAG supports logical reasoning and multi-hop fact Q&A, etc., and is significantly better than the current SOTA method.

⭐️ Star our repository to stay up-to-date with exciting new features and improvements! Get instant notifications for new releases! 🌟

![Star KAG](./_static/images/star-kag.gif)

### 1.1 Technical Architecture

![Figure 1 KAG technical architecture](./_static/images/kag-arch.png)

The KAG framework includes three parts: kg-builder, kg-solver, and kag-model. This release only involves the first two parts, kag-model will be gradually open source release in the future.

kg-builder implements a knowledge representation that is friendly to large-scale language models (LLM). Based on the hierarchical structure of DIKW (data, information, knowledge and wisdom), IT upgrades SPG knowledge representation ability, and is compatible with information extraction without schema constraints and professional knowledge construction with schema constraints on the same knowledge type (such as entity type and event type), it also supports the mutual index representation between the graph structure and the original text block, which supports the efficient retrieval of the reasoning question and answer stage.

kg-solver uses a logical symbol-guided hybrid solving and reasoning engine that includes three types of operators: planning, reasoning, and retrieval, to transform natural language problems into a problem-solving process that combines language and symbols. In this process, each step can use different operators, such as exact match retrieval, text retrieval, numerical calculation or semantic reasoning, so as to realize the integration of four different problem solving processes: Retrieval, Knowledge Graph reasoning, language reasoning and numerical calculation.

### 1.2 Knowledge Representation

In the context of private knowledge bases, unstructured data, structured information, and business expert experience often coexist. KAG references the DIKW hierarchy to upgrade SPG to a version that is friendly to LLMs. For unstructured data such as news, events, logs, and books, as well as structured data like transactions, statistics, and approvals, along with business experience and domain knowledge rules, KAG employs techniques such as layout analysis, knowledge extraction, property normalization, and semantic alignment to integrate raw business data and expert rules into a unified business knowledge graph.

![KAG Diagram](./_static/images/kag-diag.jpg)

This makes it compatible with schema-free information extraction and schema-constrained expertise construction on the same knowledge type (e. G., entity type, event type), and supports the cross-index representation between the graph structure and the original text block. This mutual index representation is helpful to the construction of inverted index based on graph structure, and promotes the unified representation and reasoning of logical forms.

### 1.3 Mixed Reasoning Guided by Logic Forms

![Logical Form Solver](./_static/images/kag-lf-solver.png)

KAG proposes a logically formal guided hybrid solution and inference engine. The engine includes three types of operators: planning, reasoning, and retrieval, which transform natural language problems into problem solving processes that combine language and notation. In this process, each step can use different operators, such as exact match retrieval, text retrieval, numerical calculation or semantic reasoning, so as to realize the integration of four different problem solving processes: Retrieval, Knowledge Graph reasoning, language reasoning and numerical calculation.

## 2. How is the effect?

### 2.1 the effect of public data sets (multi-hop reasoning)

![KAG Performance](./_static/images/kag-perf.webp)

After optimization, we not only verified the adaptability of KAG in vertical fields, but also compared it with the existing RAG method in the multi-hop Question and Answer of general data sets. We found that it is obviously better than SOTA method, with F1 increasing by 33.5% on 2wiki and 19.6% on hotpotQA. We are continuing to refine this framework and have demonstrated its effectiveness through end-to-end experiments and ablation experiment metrics. We demonstrate the validity of this framework by means of logic-symbol-driven reasoning and conceptual alignment.

### 2.2 the effect of domain knowledge scenarios (risk mining)

#### 2.2.1 Definition of Expert Rules

* definition of "gambling APP" identification rules

**define riskAppTaxo rule**

```text
Define (s:App)-[p:belongTo]->(o:`TaxOfRiskApp`/`GamblingApp`) {
Structure {
(s)
}
Constraint {
R1("risk label marked as gambling") s.riskMark like "%Gambling%"
}
}
```

* define the "App developer" identification rules
### 1.1 Key Features

The goal of KAG is to build a knowledge-enhanced LLM service framework in professional domains, supporting logical reasoning, factual Q&A, etc. KAG fully integrates the logical and factual characteristics of the KGs. Its core features include:

**define app developper rule**
- Knowledge and Chunk Mutual Indexing structure to integrate more complete contextual text information
- Knowledge alignment using conceptual semantic reasoning to alleviate the noise problem caused by OpenIE
- Schema-constrained knowledge construction to support the representation and construction of domain expert knowledge
- Logical form-guided hybrid reasoning and retrieval to support logical reasoning and multi-hop reasoning Q&A

```text
Define (s:Person)-[p:developed]->(o:App) {
Structure {
(s)-[:hasDevice]->(d:Device)-[:install]->(o)
}
Constraint {
deviceNum = group(s,o).count(d)
R1("device installed same app"): deviceNum > 5
}
}
```

* define the rules for the identification of "gambling App developers"
## 2. Latest Updates

**define a RiskUser of gambling app rule**
- 🚀 [2024-11-21] to be updated.
- 🚀 [2024-10-24] OpenSPG released v0.5, officially releasing the professional domain knowledge service framework of knowledge augmented generation (KAG).

```text
Define (s:Person)-[p:belongTo]->(o:`TaxOfRiskUser`/`DeveloperOfGamblingApp`) {
Structure {
(s)-[:developed]->(app:`TaxOfRiskApp`/`GamblingApp`)
}
Constraint {
}
}
```

#### 2.2.2 Business Data

![KAG Business Data](./_static/images/kag-biz-data.png)

#### 2.2.3 Reasoning Process

![KAG Reasoning Process](./_static/images/kag-reason.png)

The key steps in the reasoning process are as follows.

* Convert natural language problems into executable logical expressions, which rely on Conceptual Modeling under the project, and refer to the Black product mining document.

* Submit the converted logical expression to the OpenSPG reasoner for execution to obtain the user classification result.
⭐️ Star our repository to stay up-to-date with exciting new features and improvements! Get instant notifications for new releases! 🌟

* Generate answers to the user's classification results.
![Star KAG](./_static/images/star-kag.gif)

Combined with OpenSPG concept modeling, KAG can reduce the difficulty of natural language conversion graph query, change data-oriented conversion to classification concept-oriented conversion, and quickly realize the field application of natural language question and answer on the original OpenSPG project.

## 3. How to use it
## 3. Quick Start

### 3.1 product-based (for ordinary users)

Expand Down Expand Up @@ -190,6 +119,7 @@ Refer to the 3.1 section to complete the installation of the engine & dependent

Please refer to the [Quick Start](https://openspg.yuque.com/ndx6g9/wc9oyq/owp4sxbdip2u7uvv) guide for detailed introduction of the toolkit. Then you can use the built-in components to reproduce the performance results of the built-in datasets, and apply those components to new busineness scenarios.


## 4. How to extend

### 4.1 Extend KAG capability
Expand Down Expand Up @@ -277,23 +207,124 @@ The way other frameworks integrate kag can be simply described as:

![KAG integrated with other frameworks](./_static/images/kag-integrate.png)

## 5. Follow-up plan

## 5. Technical Details

### 5.1 Technical Architecture

![Figure 1 KAG technical architecture](./_static/images/kag-arch.png)

The KAG framework includes three parts: kg-builder, kg-solver, and kag-model. This release only involves the first two parts, kag-model will be gradually open source release in the future.

kg-builder implements a knowledge representation that is friendly to large-scale language models (LLM). Based on the hierarchical structure of DIKW (data, information, knowledge and wisdom), IT upgrades SPG knowledge representation ability, and is compatible with information extraction without schema constraints and professional knowledge construction with schema constraints on the same knowledge type (such as entity type and event type), it also supports the mutual index representation between the graph structure and the original text block, which supports the efficient retrieval of the reasoning question and answer stage.

kg-solver uses a logical symbol-guided hybrid solving and reasoning engine that includes three types of operators: planning, reasoning, and retrieval, to transform natural language problems into a problem-solving process that combines language and symbols. In this process, each step can use different operators, such as exact match retrieval, text retrieval, numerical calculation or semantic reasoning, so as to realize the integration of four different problem solving processes: Retrieval, Knowledge Graph reasoning, language reasoning and numerical calculation.

### 5.2 Knowledge Representation

In the context of private knowledge bases, unstructured data, structured information, and business expert experience often coexist. KAG references the DIKW hierarchy to upgrade SPG to a version that is friendly to LLMs. For unstructured data such as news, events, logs, and books, as well as structured data like transactions, statistics, and approvals, along with business experience and domain knowledge rules, KAG employs techniques such as layout analysis, knowledge extraction, property normalization, and semantic alignment to integrate raw business data and expert rules into a unified business knowledge graph.

![KAG Diagram](./_static/images/kag-diag.jpg)

This makes it compatible with schema-free information extraction and schema-constrained expertise construction on the same knowledge type (e. G., entity type, event type), and supports the cross-index representation between the graph structure and the original text block. This mutual index representation is helpful to the construction of inverted index based on graph structure, and promotes the unified representation and reasoning of logical forms.

### 5.3 Mixed Reasoning Guided by Logic Forms

![Logical Form Solver](./_static/images/kag-lf-solver.png)

KAG proposes a logically formal guided hybrid solution and inference engine. The engine includes three types of operators: planning, reasoning, and retrieval, which transform natural language problems into problem solving processes that combine language and notation. In this process, each step can use different operators, such as exact match retrieval, text retrieval, numerical calculation or semantic reasoning, so as to realize the integration of four different problem solving processes: Retrieval, Knowledge Graph reasoning, language reasoning and numerical calculation.

### 5.4 the effect of public data sets (multi-hop reasoning)

![KAG Performance](./_static/images/kag-perf.webp)

After optimization, we not only verified the adaptability of KAG in vertical fields, but also compared it with the existing RAG method in the multi-hop Question and Answer of general data sets. We found that it is obviously better than SOTA method, with F1 increasing by 33.5% on 2wiki and 19.6% on hotpotQA. We are continuing to refine this framework and have demonstrated its effectiveness through end-to-end experiments and ablation experiment metrics. We demonstrate the validity of this framework by means of logic-symbol-driven reasoning and conceptual alignment.

### 5.5 the effect of domain knowledge scenarios (risk mining)

#### 5.5.1 Definition of Expert Rules

* definition of "gambling APP" identification rules

**define riskAppTaxo rule**

```text
Define (s:App)-[p:belongTo]->(o:`TaxOfRiskApp`/`GamblingApp`) {
Structure {
(s)
}
Constraint {
R1("risk label marked as gambling") s.riskMark like "%Gambling%"
}
}
```

* define the "App developer" identification rules

**define app developper rule**

```text
Define (s:Person)-[p:developed]->(o:App) {
Structure {
(s)-[:hasDevice]->(d:Device)-[:install]->(o)
}
Constraint {
deviceNum = group(s,o).count(d)
R1("device installed same app"): deviceNum > 5
}
}
```

* define the rules for the identification of "gambling App developers"

**define a RiskUser of gambling app rule**

```text
Define (s:Person)-[p:belongTo]->(o:`TaxOfRiskUser`/`DeveloperOfGamblingApp`) {
Structure {
(s)-[:developed]->(app:`TaxOfRiskApp`/`GamblingApp`)
}
Constraint {
}
}
```

#### 5.5.2 Business Data

![KAG Business Data](./_static/images/kag-biz-data.png)

#### 5.5.3 Reasoning Process

![KAG Reasoning Process](./_static/images/kag-reason.png)

The key steps in the reasoning process are as follows.

* Convert natural language problems into executable logical expressions, which rely on Conceptual Modeling under the project, and refer to the Black product mining document.

* Submit the converted logical expression to the OpenSPG reasoner for execution to obtain the user classification result.

* Generate answers to the user's classification results.

Combined with OpenSPG concept modeling, KAG can reduce the difficulty of natural language conversion graph query, change data-oriented conversion to classification concept-oriented conversion, and quickly realize the field application of natural language question and answer on the original OpenSPG project.


## 6. Roadmap

* domain knowledge injection to realize the fusion of domain concept graph and entity graph

* kag-model optimization to improve the efficiency of KG construction and Q&A

* hallucination inhibition of knowledge logic constraints

## 6. Contact us
## 7. Contact us

**GitHub**: <https://github.com/OpenSPG/KAG>

**OpenSPG**: <https://spg.openkg.cn/>

<img src="./_static/images/openspg-qr.png" alt="Contact Us: OpenSPG QR-code" width="200">

# Cite
# 8. Citation

If you use this software, please cite it as below:

Expand Down
Binary file added _static/images/OpenSPG-1.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.