GPU utilization is 0% when the image_size=768 in training

[michaelhuang74]

When using MSCOCO dataset for training, I tried to set the image_size > 600, say 768. Then I found that the GPU utilization stayed at 0%. Following was the output of the command nvidia-smi:

Sun Dec 25 13:45:10 2016
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 367.55 Driver Version: 367.55 |
|-------------------------------+----------------------+----------------------+
| GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |
|===============================+======================+======================|
| 0 Tesla K40c On | 0000:01:00.0 Off | 0 |
| 29% 60C P0 62W / 235W | 968MiB / 11439MiB | 0% Default |
+-------------------------------+----------------------+----------------------+
| 1 Tesla K40c On | 0000:02:00.0 Off | 0 |
| 23% 39C P8 23W / 235W | 0MiB / 11439MiB | 0% Default |
+-------------------------------+----------------------+----------------------+

The command was as follows.

nohup th train.lua -style_image style/transverse.jpg -style_size 768 -image_size 768 -checkpoints_path checkpoint/ -checkpoints_name transverse.s768i768.sw10.1x -style_weight 10 -num_iterations 40000 -batch_size 1 > train_0.out &

I was using the johnson model. When the image_size <=600, there was no issue.