-
Notifications
You must be signed in to change notification settings - Fork 9
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
extended test for python implementation + adjusted exit codes for con…
…sistency
Showing
6 changed files
with
133 additions
and
43 deletions.
There are no files selected for viewing
This file was deleted.
Oops, something went wrong.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,65 @@ | ||
name: Runner CI | ||
|
||
on: | ||
push: | ||
branches: [ main ] | ||
pull_request: | ||
branches: [ main ] | ||
|
||
jobs: | ||
check-cuda: | ||
runs-on: [gpumode-nvidia-arc] | ||
timeout-minutes: 10 | ||
container: | ||
image: nvidia/cuda:12.4.0-devel-ubuntu22.04 | ||
steps: | ||
- uses: actions/checkout@v3 | ||
|
||
- name: Setup Python | ||
uses: actions/setup-python@v5 | ||
with: | ||
python-version: '3.10' | ||
|
||
- name: Install pytest | ||
shell: bash | ||
run: pip install pytest | ||
|
||
- name: Run script | ||
shell: bash | ||
run: pytest scripts/ci_test_cuda.py | ||
|
||
env: | ||
CUDA_VISIBLE_DEVICES: 0 | ||
|
||
check-pytorch: | ||
runs-on: [gpumode-nvidia-arc] | ||
timeout-minutes: 10 | ||
container: | ||
image: nvidia/cuda:12.4.0-devel-ubuntu22.04 | ||
steps: | ||
- uses: actions/checkout@v3 | ||
|
||
- name: Setup Python | ||
uses: actions/setup-python@v5 | ||
with: | ||
python-version: '3.10' | ||
|
||
- name: Install uv | ||
uses: astral-sh/setup-uv@v3 | ||
with: | ||
version: "latest" | ||
|
||
- name: Setup Python environment | ||
run: | | ||
uv venv .venv | ||
echo "VIRTUAL_ENV=$PWD/.venv" >> $GITHUB_ENV | ||
echo "$PWD/.venv/bin" >> $GITHUB_PATH | ||
uv pip install numpy torch setuptools ninja pytest | ||
- name: Run script | ||
shell: bash | ||
run: pytest scripts/ci_test_python.py | ||
|
||
env: | ||
CUDA_VISIBLE_DEVICES: 0 | ||
|
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,52 @@ | ||
import os | ||
import sys | ||
from pathlib import Path | ||
|
||
if Path().resolve().name == "scripts": | ||
os.chdir("..") | ||
|
||
sys.path.append("src/discord-cluster-manager") | ||
|
||
from leaderboard_eval import py_eval | ||
from run_eval import run_pytorch_script | ||
|
||
ref = Path("examples/identity_py/reference.py") | ||
|
||
|
||
def test_does_not_import(): | ||
# input_tt is a typo, so this won't compile | ||
sub = """ | ||
this is a syntax error | ||
""" | ||
|
||
run = run_pytorch_script(py_eval, ref.read_text(), sub, arch=None) | ||
assert run.success is False | ||
assert run.exit_code == 1 | ||
assert "IndentationError: unexpected indent\n" in run.stderr | ||
|
||
|
||
def test_error(): | ||
# no-op, runs fine but isn't correct | ||
sub = """ | ||
import torch | ||
def custom_kernel(input): | ||
return [torch.zeros_like(i) for i in input] | ||
""" | ||
run = run_pytorch_script(py_eval, ref.read_text(), sub, arch=None) | ||
assert run.success is True | ||
assert run.command == "python eval.py" | ||
# we never reach the benchmark part, because the test fails | ||
assert "warming up..." not in run.stdout | ||
assert "mismatch found! custom implementation doesnt match reference." in run.stdout | ||
assert run.exit_code == 112 | ||
assert run.result["check"] == "fail" | ||
|
||
|
||
def test_correct(): | ||
sub = Path("examples/identity_py/submission.py").read_text() | ||
|
||
run = run_pytorch_script(py_eval, ref.read_text(), sub, arch=None) | ||
assert run.success is True | ||
assert "warming up..." in run.stdout | ||
assert run.exit_code == 0 | ||
assert run.result["check"] == "pass" |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters