Skip to content

Commit

Permalink
Update scripts/decontaminate.py
Browse files Browse the repository at this point in the history
Co-authored-by: lewtun <[email protected]>
  • Loading branch information
plaguss and lewtun authored Feb 24, 2025
1 parent a35380f commit 0c68e8c
Showing 1 changed file with 1 addition and 1 deletion.
2 changes: 1 addition & 1 deletion scripts/decontaminate.py
Original file line number Diff line number Diff line change
Expand Up @@ -92,7 +92,7 @@ def build_ngram_single(document: str, ngram_size: int = 8) -> set[str]:
ngram_lookups[ds_name] = build_ngram_lookup(eval_dataset[problem_col], ngram_size=args.ngram_size)

for eval_name, ngram_lookup in ngram_lookups.items():
# Update the ngram_loopup variable for each dataset
# Update the ngram_lookup variable for each dataset
def find_contaminated(row):
# For each example we have to build the ngrams and check for all of them on each row
ngrams = build_ngram_single(row[args.problem_column], ngram_size=args.ngram_size)
Expand Down

0 comments on commit 0c68e8c

Please sign in to comment.