About environment setting

[MarkDeng1]

Hi authors,

I met an environment setting issue, when i run eval_PromptStealer.py file.

I notice that your torch version required is very outdated (pytorch 1.12.0a0+8a1a93a)， which is unavailable right now.

The error i encountered, when running eval_PromptStealer.py file:

Traceback (most recent call last):
File "eval_PromptStealer.py", line 199, in
main()
File "eval_PromptStealer.py", line 194, in main
evaluate_prompt_stealer(prompt_stealer, val_loader, save_path)
File "eval_PromptStealer.py", line 156, in evaluate_prompt_stealer
inferred_prompts, pred_modifiers = prompt_stealer.infer(images, lexica_dataset=val_loader.dataset)
File "eval_PromptStealer.py", line 69, in infer
modifiers = self.infer_modifiers(images, lexica_dataset)
File "eval_PromptStealer.py", line 85, in infer_modifiers
output_regular = Sig(self.modifier_detector(images.to(device))).cpu()
File "/fred/oz337/zdeng/promptStealing/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1553, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
File "/fred/oz337/zdeng/promptStealing/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1562, in _call_impl
return forward_call(*args, **kwargs)
File "/fred/oz337/zdeng/prompt-stealing-attack-image/src/ml_decoder/models/tresnet/tresnet.py", line 205, in forward
logits = self.head(self.embeddings)
File "/fred/oz337/zdeng/promptStealing/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1553, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
File "/fred/oz337/zdeng/promptStealing/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1562, in _call_impl
return forward_call(*args, **kwargs)
File "/fred/oz337/zdeng/prompt-stealing-attack-image/src/ml_decoder/ml_decoder/ml_decoder.py", line 167, in forward
h = self.decoder(tgt, embedding_spatial_786.transpose(0, 1)) # [embed_len_decoder, batch, 768]
File "/fred/oz337/zdeng/promptStealing/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1553, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
File "/fred/oz337/zdeng/promptStealing/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1562, in _call_impl
return forward_call(*args, **kwargs)
File "/fred/oz337/zdeng/promptStealing/lib/python3.8/site-packages/torch/nn/modules/transformer.py", line 491, in forward
seq_len = _get_seq_len(tgt, self.layers[0].self_attn.batch_first)
File "/fred/oz337/zdeng/promptStealing/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1729, in getattr
raise AttributeError(f"'{type(self).name}' object has no attribute '{name}'")
AttributeError: 'TransformerDecoderLayerOptimal' object has no attribute 'self_attn'

How do i deal with this error?

Thanks,
Mark

[MarkDeng1]

Additionally, provided modifier.pth is mismatching with tresnet_l model:

Traceback (most recent call last):
File "eval_PromptStealer.py", line 199, in
main()
File "eval_PromptStealer.py", line 192, in main
prompt_stealer = PromptStealer(subject_generator_path, modifier_detector_path, device)
File "eval_PromptStealer.py", line 29, in init
self.load_modifier_detector(modifier_detector_path)
File "eval_PromptStealer.py", line 56, in load_modifier_detector
self.modifier_detector.load_state_dict(ckpt['model'], strict=True)
File "/fred/oz337/zdeng/promptStealing/lib/python3.8/site-packages/torch/nn/modules/module.py", line 2215, in load_state_dict
raise RuntimeError('Error(s) in loading state_dict for {}:\n\t{}'.format(
RuntimeError: Error(s) in loading state_dict for TResNet:
Missing key(s) in state_dict: "head.decoder.layers.0.self_attn.in_proj_weight", "head.decoder.layers.0.self_attn.in_proj_bias", "head.decoder.layers.0.self_attn.out_proj.weight", "head.decoder.layers.0.self_attn.out_proj.bias".

[verazuo]

Hi Mark,

We require this specific PyTorch version because we need to use BLIP, ML_Decoder, and CLIP to train/test PromptStealer. This PyTorch version is compatible with the environment requirements specified by all three packages, so we recommend using it.

As mentioned in our README, NVIDIA provides a Docker environment that supports this setup, which you can refer to here: https://docs.nvidia.com/deeplearning/frameworks/pytorch-release-notes/rel_22-05.html

The size mismatch issue may also be caused by an incorrectly configured environment.

Best,
Vera