Rstudio crashes when fitting using luz

[safalabolo]

Hi,

I followed the installation instructions provided here and installed CUDA 11.3 and cuDNN 8.4 as per the support matrix.

Running the following command:

C:\Users\Utente>nvidia-smi

gives the following output:

Mon Sep 23 10:20:22 2024
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 516.01 Driver Version: 516.01 CUDA Version: 11.7 |
|-------------------------------+----------------------+----------------------+
| GPU Name TCC/WDDM | Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |
| | | MIG M. |
|===============================+======================+======================|
| 0 NVIDIA T400 WDDM | 00000000:01:00.0 On | N/A |
| 41% 53C P0 N/A / 31W | 277MiB / 2048MiB | 3% Default |
| | | N/A |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Processes: |
| GPU GI CI PID Type Process name GPU Memory |
| ID ID Usage |
|=============================================================================|
| 0 N/A N/A 9544 C+G ...5n1h2txyewy\SearchApp.exe N/A |
| 0 N/A N/A 9616 C+G C:\Windows\explorer.exe N/A |
| 0 N/A N/A 10080 C+G ...artMenuExperienceHost.exe N/A |
| 0 N/A N/A 12056 C+G ...me\Application\chrome.exe N/A |
| 0 N/A N/A 12332 C+G ...Spark\CiscoCollabHost.exe N/A |
| 0 N/A N/A 12596 C+G ...ge\Application\msedge.exe N/A |
| 0 N/A N/A 12732 C+G ...oft\OneDrive\OneDrive.exe N/A |
| 0 N/A N/A 12924 C+G ...2txyewy\TextInputHost.exe N/A |
| 0 N/A N/A 14536 C+G ...qxf38zg5c\Skype\Skype.exe N/A |
+-----------------------------------------------------------------------------+

Running the nvcc --version command shows:

nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2022 NVIDIA Corporation
Built on Tue_May__3_19:00:59_Pacific_Daylight_Time_2022
Cuda compilation tools, release 11.7, V11.7.64
Build cuda_11.7.r11.7/compiler.31294372_0

I tried running the example provided here, but RStudio crashes after running the fit part. The same issue occurs with other examples as well.

I am using CUDA, and the cuda_is_available() command returns TRUE.
Also, this simple code runs without issues:

torch_tensor(1, device = "cuda")

which outputs:

torch_tensor
1
[ CUDAFloatType{1} ]

Session Info:

sessionInfo()
R version 4.4.1 (2024-06-14 ucrt)
Platform: x86_64-w64-mingw32/x64
Running under: Windows 10 x64 (build 19045)

Matrix products: default

locale:
[1] LC_COLLATE=Italian_Italy.utf8 LC_CTYPE=Italian_Italy.utf8 LC_MONETARY=Italian_Italy.utf8 LC_NUMERIC=C LC_TIME=Italian_Italy.utf8

time zone: Europe/Rome
tzcode source: internal

attached base packages:
[1] stats graphics grDevices utils datasets methods base

other attached packages:
[1] torch_0.13.0

loaded via a namespace (and not attached):
[1] processx_3.8.4 bit_4.5.0 compiler_4.4.1 magrittr_2.0.3 cli_3.6.3 tools_4.4.1 rstudioapi_0.16.0 Rcpp_1.0.13 bit64_4.0.5
[10] coro_1.0.4 callr_3.7.6 ps_1.8.0 rlang_1.1.4

Envoroment variables:

Sys.getenv()
ALLUSERSPROFILE C:\ProgramData
APPDATA C:\Users\Utente\AppData\Roaming
CLICOLOR_FORCE 1
CommonProgramFiles C:\Program Files\Common Files
CommonProgramFiles(x86) C:\Program Files (x86)\Common Files
CommonProgramW6432 C:\Program Files\Common Files
COMPUTERNAME DESKTOP-R4FSQGB
ComSpec C:\Windows\system32\cmd.exe
CUDA_MODULE_LOADING LAZY
CUDA_PATH C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v11.7
CUDA_PATH_V11_7 C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v11.7
CURL_CA_BUNDLE C:/PROGRA1/R/R-441.1/etc/curl-ca-bundle.crt
DISPLAY :0
DriverData C:\Windows\System32\Drivers\DriverData
GFORTRAN_STDERR_UNIT -1
GFORTRAN_STDOUT_UNIT -1
GIT_ASKPASS rpostback-askpass
HOME C:\Users\Utente\Documents
HOMEDRIVE C:
HOMEPATH \Users\Utente
JD2_HOME C:\Users\Utente\AppData\Local\JDownloader 2.0
LOCALAPPDATA C:\Users\Utente\AppData\Local
LOGONSERVER \DESKTOP-R4FSQGB
MPLENGINE tkAgg
MSYS2_ENV_CONV_EXCL R_ARCH
NUMBER_OF_PROCESSORS 16
NVTOOLSEXT_PATH C:\Program Files\NVIDIA Corporation\NvToolsExt
OneDrive C:\Users\Utente\OneDrive
OneDriveConsumer C:\Users\Utente\OneDrive
ORIGINAL_XDG_CURRENT_DESKTOP undefined
OS Windows_NT
PATH C:\rtools44\x86_64-w64-mingw32.static.posix\bin;C:\rtools44\usr\bin;C:\Program
Files\R\R-4.4.1\bin\x64;C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v11.7\bin;C:\Program
Files\NVIDIA GPU Computing Toolkit\CUDA\v11.7\libnvvp;C:\Program
Files\NVIDIA\CUDNN\v9.4\bin;C:\Program Files (x86)\Common
Files\Oracle\Java\javapath;C:\Windows\system32;C:\Windows;C:\Windows\System32\Wbem;C:\Windows\System32\WindowsPowerShell\v1.0;C:\Windows\System32\OpenSSH;C:\Program
Files\PuTTY;C:\Program Files\dotnet;C:\Program Files (x86)\Windows Kits\10\Windows Performance
Toolkit;C:\Program Files\NVIDIA Corporation\Nsight Compute 2022.2.0;C:\Program Files (x86)\NVIDIA
Corporation\PhysX\Common;C:\Users\Utente\AppData\Local\Programs\Python\Python312\Scripts;C:\Users\Utente\AppData\Local\Programs\Python\Python312;C:\Users\Utente\AppData\Local\Programs\Python\Launcher;C:\Users\Utente\AppData\Local\Microsoft\WindowsApps;C:\Users\Utente\anaconda3;C:\Users\Utente\anaconda3\Scripts;C:\Users\Utente\anaconda3\condabin;C:\Program
Files\snap\bin;C:\Users\Utente.dotnet\tools;C:\Program
Files\RStudio\resources\app\bin\quarto\bin;C:\Program Files\RStudio\resources\app\bin\postback
PATHEXT .COM;.EXE;.BAT;.CMD;.VBS;.VBE;.JS;.JSE;.WSF;.WSH;.MSC
PROCESSOR_ARCHITECTURE AMD64
PROCESSOR_IDENTIFIER Intel64 Family 6 Model 167 Stepping 1, GenuineIntel
PROCESSOR_LEVEL 6
PROCESSOR_REVISION a701
ProgramData C:\ProgramData
ProgramFiles C:\Program Files
ProgramFiles(x86) C:\Program Files (x86)
ProgramW6432 C:\Program Files
PSModulePath C:\Program Files\WindowsPowerShell\Modules;C:\Windows\system32\WindowsPowerShell\v1.0\Modules
PUBLIC C:\Users\Public
PYTHONIOENCODING utf-8
R_ARCH /x64
R_CLI_HAS_HYPERLINK_IDE_HELP true
R_CLI_HAS_HYPERLINK_IDE_RUN true
R_CLI_HAS_HYPERLINK_IDE_VIGNETTE true
R_COMPILED_BY gcc 13.2.0
R_DOC_DIR C:/PROGRA1/R/R-441.1/doc
R_HOME C:/PROGRA1/R/R-441.1
R_INCLUDE_DIR C:/PROGRA1/R/R-441.1/include
R_LIBS_SITE C:/PROGRA1/R/R-441.1/site-library
R_LIBS_USER C:\Users\Utente\AppData\Local/R/win-library/4.4
R_PLATFORM
R_RTOOLS44_PATH C:\rtools44/x86_64-w64-mingw32.static.posix/bin;C:\rtools44/usr/bin
R_RUNTIME ucrt
R_SHARE_DIR C:/PROGRA1/R/R-441.1/share
R_USER C:/Users/Utente/Documents
RMARKDOWN_MATHJAX_PATH C:/Program Files/RStudio/resources/app/resources/mathjax-27
RS_LOCAL_PEER \.\pipe\11387-rsession
RS_LOG_LEVEL WARN
RS_RPOSTBACK_PATH C:/Program Files/RStudio/resources/app/bin/rpostback.exe
RS_SHARED_SECRET a6bf0920-9afa-4f41-8dd0-5a5d693ea141
RSTUDIO 1
RSTUDIO_CLI_HYPERLINKS true
RSTUDIO_CONSOLE_COLOR 256
RSTUDIO_CONSOLE_WIDTH 172
RSTUDIO_DESKTOP_EXE C:\Program Files\RStudio\rstudio.exe
RSTUDIO_MSYS_SSH C:/Program Files/RStudio/resources/app/bin/msys-ssh-1000-18
RSTUDIO_PANDOC C:/Program Files/RStudio/resources/app/bin/quarto/bin/tools
RSTUDIO_PROGRAM_MODE desktop
RSTUDIO_SESSION_PID 15484
RSTUDIO_SESSION_PORT 11387
RSTUDIO_USER_IDENTITY Utente
RSTUDIO_WINUTILS C:/Program Files/RStudio/resources/app/bin/winutils
RTOOLS40_HOME C:\RBuildTools\4.0
RTOOLS44_HOME C:\rtools44
SESSIONNAME Console
SSH_ASKPASS rpostback-askpass
SystemDrive C:
SystemRoot C:\Windows
TEMP C:\Users\Utente\AppData\Local\Temp
TERM xterm-256color
TMP C:\Users\Utente\AppData\Local\Temp
USERDOMAIN DESKTOP-R4FSQGB
USERDOMAIN_ROAMINGPROFILE DESKTOP-R4FSQGB
USERNAME Utente
USERPROFILE C:\Users\Utente
windir C:\Windows

My nvcc path:

system("which nvcc", intern = TRUE)

returns:

[1] "/c/Program Files/NVIDIA GPU Computing Toolkit/CUDA/v11.7/bin/nvcc"

Any help with diagnosing and resolving these crashes would be greatly appreciated!

[hcook01]

Did you get anywhere with this?

I am having what looks to me like essentially the same problem. All the diagnostics I have tried (such as those you show above) suggest to me that my cuda installation is correct (although I am far from an expert). But I cannot get cnn examples from the mlverse webpage to run on the gpu. R crashes every time I try to train. On the cpu they work fine.

[hcook01]

Using the brain segmentation u-net example, I have tried to isolate the problem. The crash is caused by:

model(b[[1]]$to(device = device))

where device in this case is "cuda".

I can send my model to the gpu, and I can send the masks to the gpu. But trying to send the training images to the gpu causes the crash. Any thoughts?