Incentives for lower estimates

[0x4007]

This is a draft/discussion and needs more research and ideas.

As the DAO grows and trust naturally decreases, there should be an incentive for the author and possibly assignee for them to want to make time estimates for tasks as short as realistically possible (or just as close to accurate as possible)

Otherwise the natural incentive is to over estimate all tasks to make pricing larger than it should be.

Currently the only soft incentive is the financier/admins/managers needing to approve a time estimate before "funding" tasks.

time accuracy

This is very tricky to do because even if it's a <4 hour task, in our system design it's perfectly acceptable to spread that project over a few days. Because of this, we can't simply check the timeline timestamps of when they started and when they pushed their last commit or comment to determine how long it actually took them. Finding accurate time estimates may be better suited for an LLM to be the judge but even still it might be tough to do with accuracy. RAG with all the comments in the repository may provide more clues/hints for accuracy but this still seems shaky. We could do an experiment with estimating time estimates based on RAG context and then we could consider a penalty/bonus depending on how aligned the estimate is for the LLM compared to who set the label?

It could even be revealed only when the project is closed as complete which may incentivize further the label setter to tell the truth. I.e. "accurate label set bonus reward" which can be a gradient based on how close in milliseconds the label value is to the LLM estimate.

[ubiquity-os-beta]

Note

The following contributors may be suitable for this task:

gentlementlegen

76% Match ubiquity-os-marketplace/text-conversation-rewards#166

[0x4007]

@shiv810 rfc

We could do an experiment with estimating time estimates based on RAG context and then we could consider a penalty/bonus depending on how aligned the estimate is for the LLM compared to who set the label?

I think you would be best suited for this and if we think this is the best path forward I would fund a task specifically for building this out.

[shiv810]

We could do an experiment with estimating time estimates based on RAG context and then we could consider a penalty/bonus depending on how aligned the estimate is for the LLM compared to who set the label?

I think this is a solid case for fine-tuning a model with all of our comments. The first step would be to convert the comments and accurate time estimates into the format required for fine-tuning. This would result in a model that can give reasonably accurate time estimates based on the spec and issue title. For penalties and bonuses, we could use an exponential scale, where smaller errors are less penalized, and good estimates (i.e., much lower than the LLM-produced estimate on the scale) are rewarded.

[0x4007]

Can you make a proposal for a prototype? Doesn't need to be a proper plugin. Perhaps a local cli is fastest. Just let me know what you think is best in the proposal.