Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Update evaluation scripts #37

Merged
merged 11 commits into from
Oct 29, 2024
Merged

Update evaluation scripts #37

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
11 commits
Select commit Hold shift + click to select a range
205b38d
eval update
Oct 24, 2024
560ac17
eval path
Oct 24, 2024
b0a12df
Merge branch 'main' into eval2
holgerroth Oct 28, 2024
0e2dffc
Merge branch 'main' into eval2
holgerroth Oct 29, 2024
385a359
Merge remote-tracking branch 'origin/main' into eval2
Oct 29, 2024
0427288
format
Oct 29, 2024
11b8e5d
format
Oct 29, 2024
27477aa
act
Oct 29, 2024
885feae
act
Oct 29, 2024
e621d2a
Merge remote-tracking branch 'origin/main' into eval2
Oct 29, 2024
e8858fe
act
Oct 29, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
12 changes: 11 additions & 1 deletion m3/demo/gradio_m3.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -676,6 +676,7 @@ def update_sys_message(sys_message, sv):
sv.sys_msg = sys_message
return sv


def update_modality_prompt(modality_prompt, sv):
"""Update the modality prompt"""
logger.debug(f"Updating the modality prompt")
Expand Down Expand Up @@ -785,7 +786,16 @@ def create_demo(source, model_path, conv_mode, server_port):
clear_btn.click(
fn=clear_all_convs,
inputs=[sv],
outputs=[sv, prompt_edit, chat_history, history_text, history_text_full, sys_prompt_text, model_cards_text, modality_prompt_dropdown],
outputs=[
sv,
prompt_edit,
chat_history,
history_text,
history_text_full,
sys_prompt_text,
model_cards_text,
modality_prompt_dropdown,
],
)

# States
Expand Down
4 changes: 3 additions & 1 deletion m3/eval/eval_all.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -27,7 +27,8 @@ export CONV_MODE=$3
: ${CONTAINER:?"CONTAINER env var is not set!"}
: ${DATASETS:?"DATASETS env var is not set!"}
: ${CODE:?"CODE env var is not set!"}

: ${ACCOUNT:?"ACCOUNT env var is not set!"}


sbatch eval_radvqa.slurm $MODEL_PATH $OUTPUT_FOLDER_NAME $CONV_MODE
sbatch eval_slakevqa.slurm $MODEL_PATH $OUTPUT_FOLDER_NAME $CONV_MODE
Expand All @@ -36,6 +37,7 @@ sbatch eval_pathvqa.slurm $MODEL_PATH $OUTPUT_FOLDER_NAME $CONV_MODE
sbatch eval_mimicvqa.slurm $MODEL_PATH $OUTPUT_FOLDER_NAME $CONV_MODE

sbatch eval_report_mimiccxr.slurm $MODEL_PATH $OUTPUT_FOLDER_NAME $CONV_MODE
sbatch eval_report_mimiccxr_expert.slurm $MODEL_PATH $OUTPUT_FOLDER_NAME $CONV_MODE

sbatch eval_chestxray14_class.slurm $MODEL_PATH $OUTPUT_FOLDER_NAME $CONV_MODE
sbatch eval_chexpert_class.slurm $MODEL_PATH $OUTPUT_FOLDER_NAME $CONV_MODE
Expand Down
6 changes: 3 additions & 3 deletions m3/eval/eval_chestxray14_class.slurm
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,11 +3,10 @@
#SBATCH --job-name=chestxray14_class_eval
#SBATCH --nodes=1
#SBATCH --gres=gpu:8
#SBATCH --time=2:00:00
#SBATCH --time=4:00:00
#SBATCH --partition=interactive,interactive_singlenode,grizzly,polar,polar2,polar3,polar4
#SBATCH --exclusive
#SBATCH --dependency=singleton
#SBATCH -A healthcareeng_monai

# Copyright (c) MONAI Consortium
# Licensed under the Apache License, Version 2.0 (the "License");
Expand Down Expand Up @@ -36,8 +35,9 @@ export CONV_MODE=$3
: ${CONTAINER:?"CONTAINER env var is not set!"}
: ${DATASETS:?"DATASETS env var is not set!"}
: ${CODE:?"CODE env var is not set!"}
: ${ACCOUNT:?"ACCOUNT env var is not set!"}

srun -o job_chestxray14_class_${OUTPUT_FOLDER_NAME}_eval.log --container-image=$CONTAINER \
srun -o job_chestxray14_class_${OUTPUT_FOLDER_NAME}_eval.log --container-image=$CONTAINER --acount=$ACCOUNT \
--container-mounts=$CODE:/data/code,${DATASETS}:/data/datasets,$MODEL_PATH:/data/model_path \
--no-container-mount-home \
--container-remap-root \
Expand Down
9 changes: 3 additions & 6 deletions m3/eval/eval_chestxray14_expert_class.slurm
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,10 +3,9 @@
#SBATCH --job-name=chestxray14_expert_class_eval
#SBATCH --nodes=1
#SBATCH --gres=gpu:8
#SBATCH --time=2:00:00
#SBATCH --time=4:00:00
#SBATCH --partition=interactive,interactive_singlenode,grizzly,polar,polar2,polar3,polar4
#SBATCH --exclusive
#SBATCH -A healthcareeng_monai

# Copyright (c) MONAI Consortium
# Licensed under the Apache License, Version 2.0 (the "License");
Expand Down Expand Up @@ -35,12 +34,10 @@ export CONV_MODE=$3
: ${CONTAINER:?"CONTAINER env var is not set!"}
: ${DATASETS:?"DATASETS env var is not set!"}
: ${CODE:?"CODE env var is not set!"}
: ${ACCOUNT:?"ACCOUNT env var is not set!"}

# DATASETS='/lustre/fsw/portfolios/healthcareeng/projects/healthcareeng_monai/VLM/datasets'
# TODO standardize dataset location


srun -o job_chestxray14_expert_class_${OUTPUT_FOLDER_NAME}_eval.log --container-image=$CONTAINER \
srun -o job_chestxray14_expert_class_${OUTPUT_FOLDER_NAME}_eval.log --container-image=$CONTAINER --acount=$ACCOUNT \
--container-mounts=$CODE:/data/code,${DATASETS}:/data/datasets,$MODEL_PATH:/data/model_path \
--no-container-mount-home \
--container-remap-root \
Expand Down
4 changes: 2 additions & 2 deletions m3/eval/eval_chexpert_class.slurm
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,6 @@
#SBATCH --partition=interactive,interactive_singlenode,grizzly,polar,polar2,polar3,polar4
#SBATCH --exclusive
#SBATCH --dependency=singleton
#SBATCH -A healthcareeng_monai

# Copyright (c) MONAI Consortium
# Licensed under the Apache License, Version 2.0 (the "License");
Expand Down Expand Up @@ -36,8 +35,9 @@ export CONV_MODE=$3
: ${CONTAINER:?"CONTAINER env var is not set!"}
: ${DATASETS:?"DATASETS env var is not set!"}
: ${CODE:?"CODE env var is not set!"}
: ${ACCOUNT:?"ACCOUNT env var is not set!"}

srun -o job_chexpert_class_${OUTPUT_FOLDER_NAME}_eval.log --container-image=$CONTAINER \
srun -o job_chexpert_class_${OUTPUT_FOLDER_NAME}_eval.log --container-image=$CONTAINER --acount=$ACCOUNT \
--container-mounts=$CODE:/data/code,${DATASETS}:/data/datasets,$MODEL_PATH:/data/model_path \
--no-container-mount-home \
--container-remap-root \
Expand Down
4 changes: 2 additions & 2 deletions m3/eval/eval_chexpert_expert_class.slurm
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,6 @@
#SBATCH --time=2:00:00
#SBATCH --partition=interactive,interactive_singlenode,grizzly,polar,polar2,polar3,polar4
#SBATCH --exclusive
#SBATCH -A healthcareeng_monai

# Copyright (c) MONAI Consortium
# Licensed under the Apache License, Version 2.0 (the "License");
Expand Down Expand Up @@ -35,9 +34,10 @@ export CONV_MODE=$3
: ${CONTAINER:?"CONTAINER env var is not set!"}
: ${DATASETS:?"DATASETS env var is not set!"}
: ${CODE:?"CODE env var is not set!"}
: ${ACCOUNT:?"ACCOUNT env var is not set!"}


srun -o job_chexpert_expert_class_${OUTPUT_FOLDER_NAME}_eval.log --container-image=$CONTAINER \
srun -o job_chexpert_expert_class_${OUTPUT_FOLDER_NAME}_eval.log --container-image=$CONTAINER --acount=$ACCOUNT \
--container-mounts=$CODE:/data/code,${DATASETS}:/data/datasets,$MODEL_PATH:/data/model_path \
--no-container-mount-home \
--container-remap-root \
Expand Down
4 changes: 2 additions & 2 deletions m3/eval/eval_mimicvqa.slurm
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,6 @@
#SBATCH --time=2:00:00
#SBATCH --partition=interactive,interactive_singlenode,grizzly,polar,polar2,polar3,polar4
#SBATCH --dependency=singleton
#SBATCH -A healthcareeng_monai

# Copyright (c) MONAI Consortium
# Licensed under the Apache License, Version 2.0 (the "License");
Expand Down Expand Up @@ -36,6 +35,7 @@ export CONV_MODE=$3
: ${CONTAINER:?"CONTAINER env var is not set!"}
: ${DATASETS:?"DATASETS env var is not set!"}
: ${CODE:?"CODE env var is not set!"}
: ${ACCOUNT:?"ACCOUNT env var is not set!"}

# common
export GENERATION_CONFIG='{"max_new_tokens":1024,"do_sample":false,"temperature":0}'
Expand All @@ -44,7 +44,7 @@ export GENERATION_CONFIG='{"max_new_tokens":1024,"do_sample":false,"temperature"


srun -o job_mimicvqa_${OUTPUT_FOLDER_NAME}_eval.log \
--container-image=$CONTAINER \
--container-image=$CONTAINER --acount=$ACCOUNT \
--container-mounts=$CODE:/data/code,$DATASETS:/data/datasets,$MODEL_PATH:/data/model_path \
--no-container-mount-home \
--container-remap-root \
Expand Down
4 changes: 2 additions & 2 deletions m3/eval/eval_pathvqa.slurm
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,6 @@
#SBATCH --time=2:00:00
#SBATCH --partition=interactive,interactive_singlenode,grizzly,polar,polar2,polar3,polar4
#SBATCH --dependency=singleton
#SBATCH -A healthcareeng_monai

# Copyright (c) MONAI Consortium
# Licensed under the Apache License, Version 2.0 (the "License");
Expand Down Expand Up @@ -35,12 +34,13 @@ export CONV_MODE=$3
: ${CONTAINER:?"CONTAINER env var is not set!"}
: ${DATASETS:?"DATASETS env var is not set!"}
: ${CODE:?"CODE env var is not set!"}
: ${ACCOUNT:?"ACCOUNT env var is not set!"}

# common
export GENERATION_CONFIG='{"max_new_tokens":1024,"do_sample":false,"temperature":0}'


srun -o job_pathvqa_${OUTPUT_FOLDER_NAME}_eval.log --container-image=$CONTAINER \
srun -o job_pathvqa_${OUTPUT_FOLDER_NAME}_eval.log --container-image=$CONTAINER --acount=$ACCOUNT \
--container-mounts=$CODE:/data/code,$DATASETS:/data/datasets,$MODEL_PATH:/data/model_path \
--no-container-mount-home \
--container-remap-root \
Expand Down
4 changes: 2 additions & 2 deletions m3/eval/eval_radvqa.slurm
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,6 @@
#SBATCH --time=1:00:00
#SBATCH --partition=interactive,interactive_singlenode,grizzly,polar,polar2,polar3,polar4
#SBATCH --dependency=singleton
#SBATCH -A healthcareeng_monai

# Copyright (c) MONAI Consortium
# Licensed under the Apache License, Version 2.0 (the "License");
Expand Down Expand Up @@ -35,12 +34,13 @@ export CONV_MODE=$3
: ${CONTAINER:?"CONTAINER env var is not set!"}
: ${DATASETS:?"DATASETS env var is not set!"}
: ${CODE:?"CODE env var is not set!"}
: ${ACCOUNT:?"ACCOUNT env var is not set!"}

# common
export GENERATION_CONFIG='{"max_new_tokens":1024,"do_sample":false,"temperature":0}'


srun -o job_radvqa_${OUTPUT_FOLDER_NAME}_eval.log --container-image=$CONTAINER \
srun -o job_radvqa_${OUTPUT_FOLDER_NAME}_eval.log --container-image=$CONTAINER --acount=$ACCOUNT \
--container-mounts=$CODE:/data/code,$DATASETS:/data/datasets,$MODEL_PATH:/data/model_path \
--no-container-mount-home \
--container-remap-root \
Expand Down
Loading
Loading