Adapted From the "Pruning LLMs by Weights and Activations" Paper

A Simple and Effective Pruning Approach for Large Language Models
Mingjie Sun*, Zhuang Liu*, Anna Bair, J. Zico Kolter (* indicates equal contribution)
Carnegie Mellon University, Meta AI Research and Bosch Center for AI
Paper - Project page

@article{sun2023wanda,
  title={A Simple and Effective Pruning Approach for Large Language Models}, 
  author={Sun, Mingjie and Liu, Zhuang and Bair, Anna and Kolter, J. Zico},
  year={2023},
  journal={arXiv preprint arXiv:2306.11695}
}

Acknowledgement

Their repository is built upon the SparseGPT repository.

License

Their project is released under the MIT license.

Pruning Image Classifiers

Adapted from the solution offered by the authors, which is built on the dropout repository.

Usage

Here is the command for pruning ConvNeXt/ViT models:

python main.py --model [ARCH] \
    --data_path [PATH to ImageNet] \
    --resume [PATH to the pretrained weights] \
    --prune_metric wanda \
    --prune_granularity row \
    --sparsity 0.5 

where:

• --model: network architecture, choices [convnext_base, deit_base_patch16_224, vit_base_patch16_224].
• --resume: model path to downloaded pretrained weights.
• --prune_metric: [magnitude, wanda].
• --prune_granularity: [layer, row].