Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

MMT-Bench Assertion Error #709

Open
MoyusiteruIori opened this issue Jan 4, 2025 · 0 comments
Open

MMT-Bench Assertion Error #709

MoyusiteruIori opened this issue Jan 4, 2025 · 0 comments

Comments

@MoyusiteruIori
Copy link

Command:

python run.py --data MMT-Bench_ALL_MI --model llava_next_interleave_7b --verbose

Output:

[2025-01-04 17:01:05] ERROR - misc.py: load_env - 212: Did not detect the .env file at /home/mengyu/VLMEvalKit/.env, failed to load. 
[2025-01-04 17:01:05] ERROR - misc.py: load_env - 212: Did not detect the .env file at /home/mengyu/VLMEvalKit/.env, failed to load. 
[2025-01-04 17:01:05,704] WARNING - RUN - run.py: main - 165: --reuse is not set, will not reuse previous (before one day) temporary files
[2025-01-04 17:01:05] WARNING - run.py: main - 165: --reuse is not set, will not reuse previous (before one day) temporary files
[2025-01-04 17:02:32,041] ERROR - RUN - run.py: main - 411: Model llava_next_interleave_7b x Dataset MMT-Bench_ALL_MI combination failed: , skipping this combination.
multiprocessing.pool.RemoteTraceback: 
"""
Traceback (most recent call last):
  File "/home/mengyu/miniconda3/envs/vlmevalkit/lib/python3.10/multiprocessing/pool.py", line 125, in worker
    result = (True, func(*args, **kwds))
  File "/home/mengyu/miniconda3/envs/vlmevalkit/lib/python3.10/multiprocessing/pool.py", line 48, in mapstar
    return list(map(*args))
  File "/home/mengyu/VLMEvalKit/vlmeval/smp/file.py", line 24, in decode_img_omni
    assert len(images) == len(paths)
AssertionError
"""

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/mengyu/VLMEvalKit/run.py", line 232, in main
    dataset = build_dataset(dataset_name, **dataset_kwargs)
  File "/home/mengyu/VLMEvalKit/vlmeval/dataset/__init__.py", line 207, in build_dataset
    return cls(dataset=dataset_name, **kwargs)
  File "/home/mengyu/VLMEvalKit/vlmeval/dataset/image_base.py", line 45, in __init__
    data = self.load_data(dataset)
  File "/home/mengyu/VLMEvalKit/vlmeval/dataset/image_base.py", line 143, in load_data
    return self.prepare_tsv(url, file_md5)
  File "/home/mengyu/VLMEvalKit/vlmeval/dataset/image_base.py", line 101, in prepare_tsv
    LOCALIZE(data_path, local_path)
  File "/home/mengyu/VLMEvalKit/vlmeval/tools.py", line 291, in LOCALIZE
    data_new = localize_df(data, dname)
  File "/home/mengyu/VLMEvalKit/vlmeval/smp/file.py", line 60, in localize_df
    ret = pool.map(decode_img_omni, tups)
  File "/home/mengyu/miniconda3/envs/vlmevalkit/lib/python3.10/multiprocessing/pool.py", line 367, in map
    return self._map_async(func, iterable, mapstar, chunksize).get()
  File "/home/mengyu/miniconda3/envs/vlmevalkit/lib/python3.10/multiprocessing/pool.py", line 774, in get
    raise self._value
AssertionError
[2025-01-04 17:02:32] ERROR - run.py: main - 411: Model llava_next_interleave_7b x Dataset MMT-Bench_ALL_MI combination failed: , skipping this combination.
multiprocessing.pool.RemoteTraceback: 
"""
Traceback (most recent call last):
  File "/home/mengyu/miniconda3/envs/vlmevalkit/lib/python3.10/multiprocessing/pool.py", line 125, in worker
    result = (True, func(*args, **kwds))
  File "/home/mengyu/miniconda3/envs/vlmevalkit/lib/python3.10/multiprocessing/pool.py", line 48, in mapstar
    return list(map(*args))
  File "/home/mengyu/VLMEvalKit/vlmeval/smp/file.py", line 24, in decode_img_omni
    assert len(images) == len(paths)
AssertionError
"""

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/mengyu/VLMEvalKit/run.py", line 232, in main
    dataset = build_dataset(dataset_name, **dataset_kwargs)
  File "/home/mengyu/VLMEvalKit/vlmeval/dataset/__init__.py", line 207, in build_dataset
    return cls(dataset=dataset_name, **kwargs)
  File "/home/mengyu/VLMEvalKit/vlmeval/dataset/image_base.py", line 45, in __init__
    data = self.load_data(dataset)
  File "/home/mengyu/VLMEvalKit/vlmeval/dataset/image_base.py", line 143, in load_data
    return self.prepare_tsv(url, file_md5)
  File "/home/mengyu/VLMEvalKit/vlmeval/dataset/image_base.py", line 101, in prepare_tsv
    LOCALIZE(data_path, local_path)
  File "/home/mengyu/VLMEvalKit/vlmeval/tools.py", line 291, in LOCALIZE
    data_new = localize_df(data, dname)
  File "/home/mengyu/VLMEvalKit/vlmeval/smp/file.py", line 60, in localize_df
    ret = pool.map(decode_img_omni, tups)
  File "/home/mengyu/miniconda3/envs/vlmevalkit/lib/python3.10/multiprocessing/pool.py", line 367, in map
    return self._map_async(func, iterable, mapstar, chunksize).get()
  File "/home/mengyu/miniconda3/envs/vlmevalkit/lib/python3.10/multiprocessing/pool.py", line 774, in get
    raise self._value
AssertionError

The dataset is automatically downloaded by the program. How can I fix this problem?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant