Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

DeepEval Project #67

Open
7 of 10 tasks
galshubeli opened this issue Jan 16, 2025 · 1 comment
Open
7 of 10 tasks

DeepEval Project #67

galshubeli opened this issue Jan 16, 2025 · 1 comment
Assignees
@galshubeli
Labels
medium

Comments

@galshubeli
Copy link
Contributor

galshubeli commented Jan 16, 2025 •
edited
Loading

  • Evaluate DeepEval for applicability and performance.
  • Implement custom metrics tailored for graph context evaluation.
  • Formulate additional questions to enhance evaluation criteria.
  • Define 2–4 relevant metrics for consistent evaluation.
  • Execute the evaluation pipeline across multiple models using GitHub Actions.
  • Investigate methods to compute and interpret a "soft score."
  • Benchmark using deepeval over several models
  • CI tests with threshold from the benchmark
  • Validate the evaluation of bad graphs, including prompt and ontology aspects.
  • Assess strategies for outputting scores derived from different metrics.
@barakb
Copy link

barakb commented Jan 19, 2025

@galshubeli please add checklist

@barakb barakb added the medium label Jan 19, 2025
@galshubeli galshubeli changed the title CI tests with DeepEval DeepEval project Feb 2, 2025
@galshubeli galshubeli changed the title DeepEval project DeepEval Project Feb 2, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@galshubeli galshubeli

Labels
medium
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@barakb @galshubeli