`/annotate` #67

0x4007 · 2025-01-08T11:56:41Z

There are situations where we write about certain issues that were likely to have been posted in the past, but are difficult to find.

Imagine if using vector embeddings, we can instantly link to whatever issues are being referenced:

We had some problems with KV, so I don't know if it's wise to enable plugin communication without fixing the problem (I can't find the issue about this). Alternatively we could make this as part of a module inside text-conversation-rewards

From kingsley-einstein/contributions-scan#4 (comment)

Using this command, we should replicate the behavior of issue deduplication on the existing comment and edit/add the footnotes with links to the source issues.

A limitation that I see is how wide of a search we should conduct. Within the same repository is somewhat useless, and globally might incur too much noise.

Organization wide might be the best default, but sometimes it would be very useful, especially for us using three organizations, to do a global annotation/search.

Perhaps we can have optional arguments to scope the search.

/annotate https://github.com/kingsley-einstein/contributions-scan/pull/4#issuecomment-2577468623 global

But default can be just

/annotate

And it will automatically annotate the previous comment with an organization wide search.

The text was updated successfully, but these errors were encountered:

ubiquity-os-beta · 2025-01-08T11:56:54Z

Note

The following contributors may be suitable for this task:

shiv810

78% Match ubiquity-os-marketplace/text-vector-embeddings#6
77% Match ubiquity/work.ubq.fi#119
77% Match ubiquity-os/plugins-wishlist#32

Keyrxng

77% Match ubiquity-os/plugins-wishlist#29

0x4007 · 2025-01-08T12:15:34Z

/ask I wonder if there is a more suitable command name instead of annotate which will make more sense for a global search as the default. Please provide some options.

ubiquity-os-beta · 2025-01-08T12:16:12Z

! An error occurred

ubiquity-os-beta · 2025-01-08T12:21:38Z

! No answer from OpenAI

0x4007 · 2025-01-08T12:43:20Z

/help

Keyrxng · 2025-01-15T19:48:22Z

It's interesting I think because the original plan for shiv's addition to my V1 /ask was that it was going to be accessed using /search.

/annotate - If just this string is posted, what is used as your query?
/annotate <issue.comment.url> the way you used it, you linked to foo's comment as the query I guess?

It seems to me foo or yourself should have been about to just use the /ask command to search? Not on kingsley's repo obviously but on in a Ubiquity one.

So the only truly new functionality here is the ability to annotate a particular comment with the /ask results. Now that is a great use of LLM tool calling and could be implemented v easily and activated with natural language.

koya0 · 2025-01-17T17:56:28Z

/start

ubiquity-os-beta · 2025-01-17T17:56:37Z

Beneficiary 0xd4c680d58b9c370ed810523C83B8Ec3cD66a694f

Tip

Use /wallet 0x0000...0000 if you want to update your registered payment wallet address.
Be sure to open a draft pull request as soon as possible to communicate updates on your progress.
Be sure to provide timely updates to us when requested, or you will be automatically unassigned from the task.

ubiquity-os-beta · 2025-02-05T18:55:24Z

[ 200 WXDAI ]
@koya0

Contributions Overview

View	Contribution	Count	Reward
Issue	Task	1	200

Conversation Incentives

Comment	Formatting	Relevance	Priority	Reward

[ 2.368 WXDAI ]
@Keyrxng

Contributions Overview

View	Contribution	Count	Reward
Issue	Comment	1	2.368

Conversation Incentives

Comment	Formatting	Relevance	Priority	Reward
It's interesting I think because the original plan for shiv's ad…	5.93 content: content: p: score: 0 elementCount: 5 result: 0 regex: wordCount: 122 wordValue: 0.1 result: 5.93	0.8	2	2.368

[ 146.79 WXDAI ]
@0x4007

Contributions Overview

View	Contribution	Count	Reward
Review	Base Review for #70	1	25
Review	Code Review	3	55.82
Issue	Specification	1	23.98
Review	Comment	26	41.99

Review Details for #70

Changes	Priority	Reward
+261 -6	2	5.34
+2489 -19	2	50.16
+4 -12	2	0.32

Conversation Incentives

Comment	Formatting	Relevance	Priority	Reward
There are situations where we write about certain issues that we…	11.99 content: content: p: score: 0 elementCount: 3 a: score: 5 elementCount: 1 result: 5 regex: wordCount: 148 wordValue: 0.1 result: 6.99	1	2	23.98
My review status doesn't appear to be approved so approving one …	1.54 content: content: p: score: 0 elementCount: 1 result: 0 regex: wordCount: 25 wordValue: 0.1 result: 1.54	0.2	2	0.616
Wouldn't it make more sense to have a null comment ID	0.83 content: content: p: score: 0 elementCount: 1 result: 0 regex: wordCount: 12 wordValue: 0.1 result: 0.83	0.7	2	1.162
This seems problematic if a comment contains normal footnotes. …	3.34 content: content: p: score: 0 elementCount: 1 result: 0 regex: wordCount: 62 wordValue: 0.1 result: 3.34	0.9	2	6.012
Maybe null again. Empty strings seem to be LLM code smell	0.77 content: content: p: score: 0 elementCount: 1 result: 0 regex: wordCount: 11 wordValue: 0.1 result: 0.77	0.6	2	0.924
Null or undefined makes more sense.	0.46 content: content: p: score: 0 elementCount: 1 result: 0 regex: wordCount: 6 wordValue: 0.1 result: 0.46	0.5	2	0.46
Maybe we should prefix the footnotes for issue de duplication wi…	1.33 content: content: p: score: 0 elementCount: 1 result: 0 regex: wordCount: 21 wordValue: 0.1 result: 1.33	0.8	2	2.128
```suggestionlet footnotes: string[];``&…	0 content: content: {} result: 0 regex: wordCount: 0 wordValue: 0.1 result: 0	0.4	2	0
I don't know off hand if 10 turns into 010 or just 10	0.94 content: content: p: score: 0 elementCount: 1 result: 0 regex: wordCount: 14 wordValue: 0.1 result: 0.94	0.3	2	0.564
This seems brittle to target strings like this. Any ideas to get…	1.38 content: content: p: score: 0 elementCount: 1 result: 0 regex: wordCount: 22 wordValue: 0.1 result: 1.38	0.6	2	1.656
Sure. Just seems more concise to write it that way	0.71 content: content: p: score: 0 elementCount: 1 result: 0 regex: wordCount: 10 wordValue: 0.1 result: 0.71	0.5	2	0.71
Can you make it? Seems pretty relevant and here is our first use…	0.94 content: content: p: score: 0 elementCount: 1 result: 0 regex: wordCount: 14 wordValue: 0.1 result: 0.94	0.7	2	1.316
I suppose this should be removed.	0.46 content: content: p: score: 0 elementCount: 1 result: 0 regex: wordCount: 6 wordValue: 0.1 result: 0.46	0.1	2	0.092
QA screenshot looks promising but better if you can link that th…	1.11 content: content: p: score: 0 elementCount: 1 result: 0 regex: wordCount: 17 wordValue: 0.1 result: 1.11	0.2	2	0.444
@gentlementlegen @whilefoo RFC	0.25 content: content: p: score: 0 elementCount: 1 result: 0 regex: wordCount: 3 wordValue: 0.1 result: 0.25	0.1	2	0.05
This will look up from our already generated embeddings. It need…	1.11 content: content: p: score: 0 elementCount: 1 result: 0 regex: wordCount: 17 wordValue: 0.1 result: 1.11	0.3	2	0.666
So what's the solution	0.39 content: content: p: score: 0 elementCount: 1 result: 0 regex: wordCount: 5 wordValue: 0.1 result: 0.39	0.2	2	0.156
Given this task I think we should support commands in the copilo…	6 content: content: p: score: 0 elementCount: 1 a: score: 5 elementCount: 1 result: 5 regex: wordCount: 15 wordValue: 0.1 result: 1	0.4	2	4.8
All commands move to chat box. No more in comments.	0.71 content: content: p: score: 0 elementCount: 1 result: 0 regex: wordCount: 10 wordValue: 0.1 result: 0.71	0.3	2	0.426
I had a similar problem with bun I needed to upgrade it before i…	1.22 content: content: p: score: 0 elementCount: 1 result: 0 regex: wordCount: 19 wordValue: 0.1 result: 1.22	0.4	2	0.976
https://docs.github.com/en/copilot/using-github-copilot/asking-g…	8.2 content: content: p: score: 0 elementCount: 1 a: score: 5 elementCount: 1 result: 5 regex: wordCount: 59 wordValue: 0.1 result: 3.2	0.6	2	9.84
I realized also that we could do a workaround by using the web b…	6.44 content: content: p: score: 0 elementCount: 1 img: score: 5 elementCount: 1 result: 5 regex: wordCount: 23 wordValue: 0.1 result: 1.44	0.3	2	3.864
Then comment slash command for now	0.46 content: content: p: score: 0 elementCount: 1 result: 0 regex: wordCount: 6 wordValue: 0.1 result: 0.46	0.1	2	0.092
Our plugins invoked by commands should populate on our help menu…	1.85 content: content: p: score: 0 elementCount: 1 result: 0 regex: wordCount: 31 wordValue: 0.1 result: 1.85	0.4	2	1.48
Actually I see that the manifests aren't building on push @gentl…	1.11 content: content: p: score: 0 elementCount: 1 result: 0 regex: wordCount: 17 wordValue: 0.1 result: 1.11	0.4	2	0.888
Agreed on the recommendation. @gentlementlegen maybe you can han…	2.2 content: content: p: score: 0 elementCount: 1 result: 0 regex: wordCount: 38 wordValue: 0.1 result: 2.2	0.5	2	2.2
@shiv810 @whilefoo @gentlementlegen you guys can do a retroactiv…	1.17 content: content: p: score: 0 elementCount: 1 result: 0 regex: wordCount: 18 wordValue: 0.1 result: 1.17	0.2	2	0.468

[ 38.29 WXDAI ]
@gentlementlegen

Contributions Overview

View	Contribution	Count	Reward
Review	Code Review	1	5.36
Review	Comment	9	32.93

Review Details for #70

Changes	Priority	Reward
+262 -6	2	5.36

Conversation Incentives

Comment	Formatting	Relevance	Priority	Reward
The code looks generally okay, for the lock file I think you sho…	2.49 content: content: p: score: 0 elementCount: 2 result: 0 regex: wordCount: 44 wordValue: 0.1 result: 2.49	0.9	2	4.482
It will skip the plugin only for `issue.comment_created`…	1.38 content: content: p: score: 0 elementCount: 1 result: 0 regex: wordCount: 22 wordValue: 0.1 result: 1.38	0.7	2	1.932
@0x4007 It uses this event to store the comment in the database:…	6.28 content: content: p: score: 0 elementCount: 1 a: score: 5 elementCount: 1 result: 5 regex: wordCount: 20 wordValue: 0.1 result: 1.28	0.8	2	10.048
@0x4007 do you have access to it on mobile devices? I don't seem…	1.22 content: content: p: score: 0 elementCount: 1 result: 0 regex: wordCount: 19 wordValue: 0.1 result: 1.22	0.1	2	0.244
@koya0 You have to update `bun.lockb` because it is not …	1.06 content: content: p: score: 0 elementCount: 1 result: 0 regex: wordCount: 16 wordValue: 0.1 result: 1.06	0.8	2	1.696
@koya0 You can try upgrading bun to `1.2` where lock fil…	1.28 content: content: p: score: 0 elementCount: 1 result: 0 regex: wordCount: 20 wordValue: 0.1 result: 1.28	0.75	2	1.92
@koya0 Can't you just delete it now that you are using the plain…	1.06 content: content: p: score: 0 elementCount: 1 result: 0 regex: wordCount: 16 wordValue: 0.1 result: 1.06	0.4	2	0.848
I confirm it is available in the browser on mobile versions. But…	7.92 content: content: p: score: 0 elementCount: 3 a: score: 5 elementCount: 1 result: 5 regex: wordCount: 53 wordValue: 0.1 result: 2.92	0.5	2	7.92
It is normal that the manifest is not built, there are no push e…	3.2 content: content: p: score: 0 elementCount: 1 result: 0 regex: wordCount: 59 wordValue: 0.1 result: 3.2	0.6	2	3.84

[ 60.21 WXDAI ]
@shiv810

Contributions Overview

View	Contribution	Count	Reward
Review	Code Review	1	55.18
Review	Comment	2	5.03

Review Details for #70

Changes	Priority	Reward
+2742 -17	2	55.18

Conversation Incentives

Comment

Formatting

Relevance

Priority

Reward

This regex will match all footnotes, i.e., any `[^<NUM>…

1.7

content:
  content:
    p:
      score: 0
      elementCount: 1
  result: 0
regex:
  wordCount: 28
  wordValue: 0.1
  result: 1.7

0.8

2

2.72

We could something like this, this should be better than handlin…

1.65

content:
  content:
    p:
      score: 0
      elementCount: 1
  result: 0
regex:
  wordCount: 27
  wordValue: 0.1
  result: 1.65

0.7

2

2.31

[ 63.024 WXDAI ]
@whilefoo

Contributions Overview

View	Contribution	Count	Reward
Review	Code Review	1	55.18
Review	Comment	5	7.844

Review Details for #70

Changes	Priority	Reward
+2742 -17	2	55.18

Conversation Incentives

Comment	Formatting	Relevance	Priority	Reward
Using metadata would be more reliable, you'd need to use `po…	1.54 content: content: p: score: 0 elementCount: 1 result: 0 regex: wordCount: 25 wordValue: 0.1 result: 1.54	0.8	2	2.464
The kernel should be modified so that if the plugin manifest def…	2.15 content: content: p: score: 0 elementCount: 1 result: 0 regex: wordCount: 37 wordValue: 0.1 result: 2.15	0.6	2	2.58
This is an additional way to trigger commands, right? You'd stil…	2.2 content: content: p: score: 0 elementCount: 1 result: 0 regex: wordCount: 38 wordValue: 0.1 result: 2.2	0.4	2	1.76
I think on mobile it's only for Github Pro users	0.77 content: content: p: score: 0 elementCount: 1 result: 0 regex: wordCount: 11 wordValue: 0.1 result: 0.77	0.2	2	0.308
Normally if there's a conflict with `bun.lockb`, you jus…	1.22 content: content: p: score: 0 elementCount: 1 result: 0 regex: wordCount: 19 wordValue: 0.1 result: 1.22	0.3	2	0.732

0x4007 · 2025-02-05T19:01:39Z

Hey @koya0 I was trying to use it here any idea why it's not working

ubiquity-os/plugins-wishlist#66 (comment)

koya0 · 2025-02-05T19:11:02Z

It could be that no similar issues were found. Is there a specific issue you wanted the command to make a reference to? I could test it in my repo and see if this's what its really happening

koya0 · 2025-02-05T19:11:41Z

or maybe you could make a comment exactly like the body of the issue and try the command

koya0 · 2025-02-05T19:15:42Z

or maybe you could make a comment exactly like the body of the issue and try the command

i tried this here and worked: koya0/.ubiquity-os#32

0x4007 · 2025-02-05T20:16:18Z

or maybe you could make a comment exactly like the body of the issue and try the command

Is there a bottom threshold? Can strings match even with a 1% match? I think that might be why it's not showing anything.

@shiv810 maybe you have some perspective on how the matching works exactly with issue deduplication but the string I wanted annotated was

There is a GitHub issue up to predict time labels.

ubiquity-os-marketplace/daemon-pricing#71

koya0 · 2025-02-05T20:22:22Z

Is there a bottom threshold? Can strings match even with a 1% match? I think that might be why it's not showing anything.

the command only identifies similar issues above or equal the warning.Threshold, but it considers the whole comment body

0x4007 · 2025-02-05T20:49:34Z

warning.Threshold from issue deduplication? It should have its own config properties surely! We should make another task for it.

koya0 · 2025-02-05T20:51:08Z

i made some tests with koya0/.ubiquity-os#33 (comment) and apparently the comment you wanted to annotate is 84% similar to the issue of the own comment, so at least the bot should annotate that issue

As the supabase function im using to get similar issues its probably not the same of the prod supabase, I don't know if in prod the comment is not that similar to the issue or the command really has a problem

koya0 · 2025-02-05T20:52:13Z

is it possible to check the bot log? If yes, we can confirm what is the problem

0x4007 · 2025-02-05T20:57:02Z

is it possible to check the bot log? If yes, we can confirm what is the problem

Possibly here https://github.com/ubiquity-os-marketplace/text-vector-embeddings/actions/workflows/compute.yml

koya0 · 2025-02-05T21:09:30Z

I couldn't find the comments you made with the annotate command

koya0 · 2025-02-05T21:10:58Z

/annotate

koya0 · 2025-02-05T21:12:10Z

/annotate

for this one no similar issues were found https://github.com/ubiquity-os-marketplace/text-vector-embeddings/actions/runs/13166645094/job/36748291657

0x4007 · 2025-02-05T21:13:19Z

Maybe the action is only used to store the embedding I'm not sure since I never worked on this repo. You should know better where the logs would appear I would guess!

shiv810 · 2025-02-05T21:39:52Z

Is there a bottom threshold? Can strings match even with a 1% match? I think that might be why it's not showing anything.

Yes, by default all the embeddings would have a cosine similarity value more than 0.5.

/annotate

This approach won't work because context.command would be undefined. The issue should be addressed in the manifest.json file. Alternatively, instead of checking context.command, we should check for /annotate.

text-vector-embeddings/src/plugin.ts

Lines 36 to 38 in 0f91884

    
           if (context.command) { 
        
             return await commandHandler(context); 
        
           }

koya0 · 2025-02-05T21:46:01Z

Is there a bottom threshold? Can strings match even with a 1% match? I think that might be why it's not showing anything.

Yes, by default all the embeddings would have a cosine similarity value more than 0.5.

/annotate

This approach won't work because context.command would be undefined. The issue should be addressed in the manifest.json file. Alternatively, instead of checking context.command, we should check for /annotate.

text-vector-embeddings/src/plugin.ts

Lines 36 to 38 in 0f91884

if (context.command) {
return await commandHandler(context);
}

when a comment is created I call userAnnotate, that checks the /annotate prefix and runs the command

text-vector-embeddings/src/plugin.ts

Lines 40 to 44 in 0f91884

    
           if (isIssueCommentEvent(context)) { 
        
             switch (eventName) { 
        
               case "issue_comment.created": 
        
                 await addComments(context as Context<"issue_comment.created">); 
        
                 return await userAnnotate(context as Context<"issue_comment.created">);

shiv810 · 2025-02-05T22:00:04Z

when a comment is created I call userAnnotate, that checks the /annotate prefix and runs the command

The kernel will not forward this command unless it's invoked through one of two methods: either @UbiquityOS .... or /annotate. For the command to work, it must be defined in the manifest.json. Otherwise, the event will not be forwarded.

@whilefoo rfc ?

koya0 · 2025-02-05T22:12:28Z

when a comment is created I call userAnnotate, that checks the /annotate prefix and runs the command

The kernel will not forward this command unless it's invoked through one of two methods: either @UbiquityOS .... or /annotate. For the command to work, it must be defined in the manifest.json. Otherwise, the event will not be forwarded.

@whilefoo rfc ?

but this command: #67 (comment) worked. If you check Execute plugin of https://github.com/ubiquity-os-marketplace/text-vector-embeddings/actions/runs/13166645094/job/36748291657, in the end, there is a message saying that no similar issues were found for the comment, generated by the annotate function:

text-vector-embeddings/src/handlers/annotate.ts

Line 69 in 0f91884

context.logger.info("No similar issues found for comment", { commentBody });

shiv810 · 2025-02-05T22:31:30Z

but this command: #67 (comment) worked. If you check Execute plugin of https://github.com/ubiquity-os-marketplace/text-vector-embeddings/actions/runs/13166645094/job/36748291657, in the end, there is a message saying that no similar issues were found for the comment, generated by the annotate function:

I am not sure how annotate.ts works, but the line you are pointing to would indicate, that the input comment was I couldn't find the comments you made with the annotate command and not /annotate ....

koya0 · 2025-02-05T22:35:13Z

but this command: #67 (comment) worked. If you check Execute plugin of https://github.com/ubiquity-os-marketplace/text-vector-embeddings/actions/runs/13166645094/job/36748291657, in the end, there is a message saying that no similar issues were found for the comment, generated by the annotate function:

I am not sure how annotate.ts works, but the line you are pointing to would indicate, that the input comment was I couldn't find the comments you made with the annotate command and not /annotate ....

I couldn't find the comments you made with the annotate command is the comment that I wanted to get similar issues from, when you comment /annotate, the command should find similar issues from the previous comment made

whilefoo · 2025-02-06T12:54:08Z

The kernel will not forward this command unless it's invoked through one of two methods: either @UbiquityOS .... or /annotate. For the command to work, it must be defined in the manifest.json. Otherwise, the event will not be forwarded.

@whilefoo rfc ?

If listeners include issue_comment.created and there's no command in the manifest.json, it will send the event anyway but if there's a command defined then it won't send it unless the command matches.

So it seems the plugin is ran but it doesn't find any similar issues

0x4007 · 2025-02-06T13:46:34Z

So it seems the plugin is ran but it doesn't find any similar issues

We'll definitely need an output of sorts. I wish there was an appropriate emoji reaction to the annotate comment but not sure there is.

It can post an error message in the form of a new comment explaining that it ran successfully but there's no good matches.

0x4007 added Time: <4 Hours Priority: 2 (Medium) labels Jan 8, 2025

ubiquity-os-beta bot added the Price: 200 USD label Jan 8, 2025

devpool-directory-superintendent bot mentioned this issue Jan 8, 2025

/annotate ubiquity/devpool-directory#2022

Closed

gentlementlegen mentioned this issue Jan 8, 2025

The command /ask fails on large contexts ubiquity-os-marketplace/command-ask#48

Closed

ubiquity-os-beta bot assigned koya0 Jan 17, 2025

koya0 mentioned this issue Jan 17, 2025

feat: implement annotate command #70

Merged

0x4007 closed this as completed in #70 Feb 5, 2025

0x4007 mentioned this issue Feb 5, 2025

Annotate config #79

Open

0x4007 mentioned this issue Feb 6, 2025

No annotate matches error message #81

Open

/annotate #67

/annotate #67

Comments

0x4007 commented Jan 8, 2025 • edited Loading

ubiquity-os-beta bot commented Jan 8, 2025 • edited Loading

0x4007 commented Jan 8, 2025

ubiquity-os-beta bot commented Jan 8, 2025

ubiquity-os-beta bot commented Jan 8, 2025

0x4007 commented Jan 8, 2025

Keyrxng commented Jan 15, 2025

koya0 commented Jan 17, 2025

ubiquity-os-beta bot commented Jan 17, 2025

ubiquity-os-beta bot commented Feb 5, 2025 • edited Loading

Contributions Overview

Conversation Incentives

Contributions Overview

Conversation Incentives

Contributions Overview

Review Details for #70

Conversation Incentives

Contributions Overview

Review Details for #70

Conversation Incentives

Contributions Overview

Review Details for #70

Conversation Incentives

Contributions Overview

Review Details for #70

Conversation Incentives

0x4007 commented Feb 5, 2025

koya0 commented Feb 5, 2025

koya0 commented Feb 5, 2025

koya0 commented Feb 5, 2025

0x4007 commented Feb 5, 2025 • edited Loading

koya0 commented Feb 5, 2025 • edited Loading

0x4007 commented Feb 5, 2025

koya0 commented Feb 5, 2025

koya0 commented Feb 5, 2025

0x4007 commented Feb 5, 2025

koya0 commented Feb 5, 2025

koya0 commented Feb 5, 2025

koya0 commented Feb 5, 2025 • edited Loading

0x4007 commented Feb 5, 2025

shiv810 commented Feb 5, 2025

koya0 commented Feb 5, 2025 • edited Loading

shiv810 commented Feb 5, 2025

koya0 commented Feb 5, 2025

shiv810 commented Feb 5, 2025 • edited Loading

koya0 commented Feb 5, 2025

whilefoo commented Feb 6, 2025

0x4007 commented Feb 6, 2025

`/annotate` #67

`/annotate` #67

0x4007 commented Jan 8, 2025 •

edited

Loading

ubiquity-os-beta bot commented Jan 8, 2025 •

edited

Loading

ubiquity-os-beta bot commented Feb 5, 2025 •

edited

Loading

0x4007 commented Feb 5, 2025 •

edited

Loading

koya0 commented Feb 5, 2025 •

edited

Loading

koya0 commented Feb 5, 2025 •

edited

Loading

koya0 commented Feb 5, 2025 •

edited

Loading

shiv810 commented Feb 5, 2025 •

edited

Loading