Skip to content

Commit

Permalink
reorganize headers
Browse files Browse the repository at this point in the history
  • Loading branch information
vxw3t8fhjsdkghvbdifuk committed Jan 21, 2025
1 parent 6045d9d commit 02b3e30
Showing 1 changed file with 10 additions and 11 deletions.
21 changes: 10 additions & 11 deletions docs/source/quicktour.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -20,24 +20,23 @@ Lighteval can be used with a few different commands.
- `tgi`: evaluate models on one or more GPUs using [🔗 Text Generation Inference](https://huggingface.co/docs/text-generation-inference/en/index)
- `openai`: evaluate models on one or more GPUs using [🔗 OpenAI API](https://platform.openai.com/)

## Accelerate
## Basic usage

### Evaluate a model on a GPU

To evaluate `GPT-2` on the Truthful QA benchmark, run:
To evaluate `GPT-2` on the Truthful QA benchmark with [🤗
Accelerate](https://github.com/huggingface/accelerate) , run:

```bash
lighteval accelerate \
"pretrained=gpt2" \
"leaderboard|truthfulqa:mc|0|0"
```

Here, the first argument specifies the model to run, and the second argument specifies which tasks to run.
Here, we first choose a backend (either `accelerate`, `nanotron`, or `vllm`), and then specify the model and task(s) to run.

The syntax for the model arguments is `key1=value1,key2=value2,etc`.
The keys correspond with the backend configuration (accelerate, vllm), and are detailed [below](#Model Arguments).
Valid key-value pairs correspond with the backend configuration, and are detailed [below](#Model Arguments).

The syntax for the task specification might be a bit hard to grasp as first. The format is as follows:
The syntax for the task specification might be a bit hard to grasp at first. The format is as follows:

```txt
{suite}|{task}|{num_few_shot}|{0 for strict `num_few_shots`, or 1 to allow a reduction}
Expand Down Expand Up @@ -65,7 +64,7 @@ lighteval accelerate \
# or, e.g., "leaderboard|truthfulqa:mc|0|0|,leaderboard|gsm8k|3|1"
```

### Evaluate a model on one or more GPUs
## Evaluate a model on one or more GPUs

#### Data parallelism

Expand Down Expand Up @@ -104,13 +103,13 @@ This will automatically use accelerate to distribute the model across the GPUs.
> `model_parallel=True` and using accelerate to distribute the data across the
GPUs.

### Model Arguments
## Backend configuration

The `model-args` argument takes a string representing a list of model
argument. The arguments allowed vary depending on the backend you use (vllm or
accelerate).

#### Accelerate
### Accelerate

- **pretrained** (str):
HuggingFace Hub model ID name or the path to a pre-trained
Expand Down Expand Up @@ -146,7 +145,7 @@ accelerate).
- **trust_remote_code** (bool): Whether to trust remote code during model
loading.

#### VLLM
### VLLM

- **pretrained** (str): HuggingFace Hub model ID name or the path to a pre-trained model to load.
- **gpu_memory_utilisation** (float): The fraction of GPU memory to use.
Expand Down

0 comments on commit 02b3e30

Please sign in to comment.