Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[product documentation] experiment with a "highlight" summarizer #205921

Closed
pgayvallet opened this issue Jan 8, 2025 · 1 comment · Fixed by #206578
Closed

[product documentation] experiment with a "highlight" summarizer #205921

pgayvallet opened this issue Jan 8, 2025 · 1 comment · Fixed by #206578
Labels
Feature:AI Product Docs Product Documentation for AI workflows Team:AI Infra AppEx AI Infrastructure Team

Comments

@pgayvallet
Copy link
Contributor

semantic_text will support highlight in 8.18 / 9.0. We should experiment with an "highlight" based summarized, that could replace the current "llm summarizer" we're currently using.

Even if probably less powerful, the upside is that it would be very significantly faster than calling the LLM for summarization, which could make a great default.

kibana/x-pack/platform/plugins/shared/ai_infra/llm_tasks/server/tasks/retrieve_documentation/retrieve_documentation.ts

Lines 52 to 63 in a0f5a7f

if (tokenReductionStrategy === 'summarize') {
const extractResponse = await summarizeDocument({
searchTerm,
documentContent: document.content,
outputAPI,
connectorId,
functionCalling,
});
content = truncate(extractResponse.summary, maxDocumentTokens);
} else {
content = truncate(document.content, maxDocumentTokens);
}

@pgayvallet pgayvallet added Feature:AI Product Docs Product Documentation for AI workflows Team:AI Infra AppEx AI Infrastructure Team labels Jan 8, 2025
@elasticmachine
Copy link
Contributor

Pinging @elastic/appex-ai-infra (Team:AI Infra)

This was referenced Jan 14, 2025
Merged
Open
kibanamachine pushed a commit to kibanamachine/kibana that referenced this issue Jan 16, 2025
@pgayvallet
[product doc] implement highlight summarizer (elastic#206578)
0dfafed 
## Summary

Fix elastic#205921

- Implements a new summary strategy for the product documentation, based
on `semantic_text` highlights
- set that new strategy as the default one

### Why ?

Until now, in case of excessive token count, we were using a LLM based
summarizer. Realistically, highlights will always be worse than calling
a LLM for a "in context summary", but from my testing, highlights seem
"good enough", and the speed difference (instant for highlights vs
multiple seconds, up to a dozen, for the LLM summary) is very
significant, and seems overall worth it.

The main upside with that change, given that requesting the product doc
will be waaaay faster, is that we can then tweak the assistant's
instruction to more aggressively call the product_doc tool between each
user message without the risk of the user experience being impacted
(waiting way longer between messages). - *which will be done as a
follow-up*

### How to test ?

Install the product doc, ask questions to the assistant, check the tool
calls (sorry, don't have a better option atm...)

Note: that works with both versions of the product doc artifacts, so
don't need the dev repository

(cherry picked from commit c9286ec)
viduni94 pushed a commit to viduni94/kibana that referenced this issue Jan 23, 2025
@pgayvallet @viduni94
[product doc] implement highlight summarizer (elastic#206578)
1265289 
## Summary

Fix elastic#205921

- Implements a new summary strategy for the product documentation, based
on `semantic_text` highlights
- set that new strategy as the default one

### Why ?

Until now, in case of excessive token count, we were using a LLM based
summarizer. Realistically, highlights will always be worse than calling
a LLM for a "in context summary", but from my testing, highlights seem
"good enough", and the speed difference (instant for highlights vs
multiple seconds, up to a dozen, for the LLM summary) is very
significant, and seems overall worth it.

The main upside with that change, given that requesting the product doc
will be waaaay faster, is that we can then tweak the assistant's
instruction to more aggressively call the product_doc tool between each
user message without the risk of the user experience being impacted
(waiting way longer between messages). - *which will be done as a
follow-up*

### How to test ?

Install the product doc, ask questions to the assistant, check the tool
calls (sorry, don't have a better option atm...)

Note: that works with both versions of the product doc artifacts, so
don't need the dev repository
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
Feature:AI Product Docs Product Documentation for AI workflows Team:AI Infra AppEx AI Infrastructure Team
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

[product doc] implement highlight summarizer pgayvallet/kibana
2 participants
@pgayvallet @elasticmachine