Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters. This flag results in an extra traversal of the autograd graph every iteration, which can adversely affect performance.

[athenawisdoms]

Hi, I started noticing the following warning message after setting up a new conda environment with Pytorch 1.8.1, which is an update from my previous environment that uses Pytorch 1.7.0.

Epoch 0: 0%|
[W reducer.cpp:1050] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters. This flag results in an extra traversal of the autograd graph every iteration, which can adversely affect performance. If your model indeed never has any unused parameters, consider turning this flag off. Note that this warning may be a false positive your model has flow control causing later iterations to have unused parameters. (function operator())

Any idea if this is a real concern? How can we disable find_unused_parameters?

trainer = pl.Trainer(
    val_check_interval=0.1,
    gpus=-1,
    accelerator="ddp",
    callbacks=[checkpoint_callback, early_stop_callback],
    precision=16,
)

Packages:

• pytorch 1.8.1
• pytorch-lightning 1.2.6
• cudatoolkit 11.1.1
• cudnn 8.0.5
• python 3.8

[ananthsub]

Hi @athenawisdoms the docs here cover how you can disable find_unused_parameters and speed up your DDP training

https://pytorch-lightning.readthedocs.io/en/latest/benchmarking/performance.html#when-using-ddp-set-find-unused-parameters-false

[tong-zeng]

I was trying to import the DDPStrategy according to the documentation (with the link above):

from pytorch_lightning.strategies import DDPStrategy

However, I got the error ModuleNotFoundError: No module named 'pytorch_lightning.strategies' in pl version '1.5.8' and '1.5.10'

It seems the pytorch_lightning.strategies are not exposed to users. Is the documentation out of date?

[divanoresia]

I also encountered this problem, how did you solve it?

[ayansengupta17]

The link doesn't exist anymore

[surajiitd]

https://pytorch-lightning.readthedocs.io/en/1.3.8/benchmarking/performance.html#when-using-ddp-set-find-unused-parameters-false

[ayansengupta17]

Note that the above link is for version 1.3.8

[rohan-varma]

Hi! I added this warning in native PyTorch as a way to remind users to disable this flag if performance is critical and there are no unused parameters. One note is - as the warning indicates it can be a false positive if your model has flow control that results in unused params in future iterations.

[Bannng]

Hi, I'm curious of meaning of 'performance' in your warning sign,
does it mean the model performance or training (train time)?

[AntreasAntoniou]

I believe it refers to update time needed, since that flag enables more checks during gradient computation

[kingmbc]

Hi. The warning could disappear by adding plugins as below.

from pytorch_lightning.plugins import DDPPlugin

trainer = pl.Trainer(
    val_check_interval=0.1,
    gpus=-1,
    accelerator="ddp",
    callbacks=[checkpoint_callback, early_stop_callback],
    plugins=DDPPlugin(find_unused_parameters=False),
    precision=16,
)

[loretoparisi]

So in my case I had that warning and using

trainer = pl.Trainer(max_epochs=hparams["epochs"],
                         devices=devices,
                         accelerator=accelerator,
                         strategy='ddp')

where

accelerator = 'gpu' if torch.cuda.is_available() else 'cpu'

so how to change it using the DDPPlugin plugin?

[iamstarlee]

If it is pytorch how to solve it ? I got this issue with ''2.3.0+cu121''

[Ending2015a]

Hello. For those who are using pl 1.6.0, use DDPStrategy instead of DDPPlugin, which is deprecated after 1.6.0:

from pytorch_lightning.strategies.ddp import DDPStrategy
trainer = pl.Trainer(
  strategy = DDPStrategy(find_unused_parameters=False),
  accelerator = 'gpu',
  devices = 3
)

[akihironitta]

@Ending2015a You can use this alias for that so there's no need to import DDPStrategy just to turn it off :)

Trainer(strategy="ddp_find_unused_parameters_false", accelerator="gpu", devices="auto")

[claytonEtillman]

@akihironitta Thanks for the tip. However, when I set that strategy in my Trainer, I then receive the following output:

Global seed set to 42
GPU available: True (cuda), used: True
TPU available: False, using: 0 TPU cores
IPU available: False, using: 0 IPUs
HPU available: False, using: 0 HPUs
Number of parameters in network: 12742.1k
INFO: Will watch for changes in these directories: ['/home/ubuntu/StockTips/api/train']
ERROR: [Errno 98] Address already in use
INFO: Will watch for changes in these directories: ['/home/ubuntu/StockTips/api/train']
ERROR: [Errno 98] Address already in use
INFO: Will watch for changes in these directories: ['/home/ubuntu/StockTips/api/train']
ERROR: [Errno 98] Address already in use
Global seed set to 42
Initializing distributed: GLOBAL_RANK: 0, MEMBER: 1/4

It fails to continue after that and just hangs. Notice the global seed being set a second time although it's only set once in my code. It's as if it's restarting the api I'm using to call the Trainer.fit function.

[akihironitta]

ERROR: [Errno 98] Address already in use

Hi @claytonEtillman! The error message seems irrelevant to the warning described by the author of this discussion. Would you mind opening a new discussion?