Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

added calendar-arithmetic tasks #43

Merged
merged 1 commit into from
Feb 2, 2025
Merged

Conversation

bjiro
Copy link
Collaborator

@bjiro bjiro commented Feb 2, 2025

Added 7 tasks for calendar arithmetic.

For prompt diversity, each has 3 question templates and they share different format requests selected at random (eg: specify if the expected answer is a day in letters or a numerical value).

The calendar arithmetic problems (generally) span a year, specified in the config. The tasks have different inherent complexities, with three parameters that can be tuned to adjust the difficulty:

Tunable:
"is_leap_year" - tuned with leap_year_range

The next four tasks difficulty depend on offset_upper_bound. They can be easy to hard, by setting the offset bound from a few days to an arbitrary large number.
"weekday_offset"
"count_business_days"
"count_days"
"weekday_of_date_from_first_day" (offset is capped to fit in a year)

Not tunable (and hard, especially if given year is not in the training distribution of the llm):
"weekday_of_date"
"recurring_event_day" (harder)

I also created a denser reward for numerical answers and for strings (eg: wrong capitalization shouldn't have 0 reward).

Copy link

github-actions bot commented Feb 2, 2025

❌ Pre-commit checks failed. Please run pre-commit run --all-files locally and fix the issues.

Copy link
Contributor

@andreaskoepf andreaskoepf left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Great contribution! Thank you very much!

@andreaskoepf andreaskoepf merged commit aa172a1 into main Feb 2, 2025
2 of 3 checks passed
@andreaskoepf andreaskoepf linked an issue Feb 2, 2025 that may be closed by this pull request
@andreaskoepf andreaskoepf deleted the calendar-arithmetic branch February 25, 2025 09:24
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Add weekday claculation dataset
2 participants