Setting up a CUDA project on a remote server

Answered

Hi,

I set up the server and cmake toolkits etc. but the Debugger which automatically runs, throws an error saying 

CMake Error at /Applications/CLion.app/Contents/bin/cmake/mac/share/cmake-3.21/Modules/CMakeDetermineCUDACompiler.cmake:113 (message):
  Could not find nvcc executable in path specified by
CUDAToolkit_ROOT=/<PATH TO MY HOME DIR>/cuda/bin/

I checked some previous posts but I can't set the /etc/environment file as I do not have sudo permission.

I've tried setting these 2 variables with many different values

set(CMAKE_CUDA_COMPILER /<HOME DIR PATH>/cuda/bin/nvcc)
set(CUDAToolkit_ROOT /<HOME DIR PATH>/cuda/bin/)

I'm able to build and run the project without any errors though.

EDIT:

I'm using Mac OS and running CUDA remotely on a server. Almost everything works except the debugger.

14 comments
Comment actions Permalink

TLDR:

install from snap with snap install clion --classic, not from flatpak

maybe launch clion with exec=bash -i -c "....."

 

I had this same problem. I couldn't find a solution anywhere.

I had nvcc on my PATH, set in /etc/environment (as suggested by jetbrains). I configured the CUDAToolkit_ROOT as indicated in the clion docs. I tried setting various paths to nvcc in my project CMakeLists.txt, or in the settings->Build, Execution, Deployment->CMake flags, I tried specifying paths manually under settings->Build, ...->Toolchains - no joy. CMake couldn't find nvcc.

Of course, building the project from terminal worked just fine. It was Clion whom had mucked something royally up. I wanted to use an IDE to debug the project itself (joke's on me), so I persisted.

Clion and Pycharm have a history of clearing and misconfiguring you system environment while launching the application (in my experience - on Ubuntu). Don't ask me how or why or where - I hate this configuration stuff instead of doing actual work, that's why I use an IDE (joke's on me again).

The first hint was under settings->...->Toolchains. I configured CMake: /usr/bin/cmake (my system install). Underneath the godforsaken setting box, Clion said:

(checkmark) Version: 3.22.something

But in my system terminal, I do:

/usr/bin/cmake --version

and get:

cmake version 3.16.3

So Clion was lying, and was still using the built-in CMake version bundled with the Clion application package (somewhere under /var/lib/flatpak/app/com.jetbrains.something/something/bin/something). At this point, everything about setting cmake build flags and environment variables goes out the window. Who knows where those configurations were or weren't applied?

Anyhow, I had had previous success with Pycharm doing a similar environment-clearing-and-buggy behavior. I had gone to the .desktop file which launched Pycharm (under Ubuntu-gnome), and edited the line:

Exec="..."

to be:

Exec=bash -i -c "..."

which would magically preserve my environment variables in Pycharm (the behavior around env vars is still buggy in these IDEs, to be clear).

This fix did not apply to my situation here in 2022 with Clion - flatpak has a .desktop file somewhere deep within its inscrutable path, but the bash -i -c option had no effect here.

I uninstalled the flatpak Clion, and installed the snap version. The Ubuntu software center is too dumb to manage this - because Clion requires classic confinement (wow cool).

To install then, I did:

sudo snap install clion --classic

and magically everything works as expected.

 

Side note: I’ve also had to put shared-object files built by my projects (*.so) directly onto the system LD_LIBRARY_PATH in order for Pycharm/Clion to find them. Good luck with that. 
















0
Comment actions Permalink

Thanks for the detailed reply Chenn.

I should've mentioned that I'm using Mac and I'm running things remotely on a server (edited my question now). I installed CLion by simply downloading it from jetbrains and not via flatpak or snap.

I'm manually pointing it to all the CUDA libraries, CMAKE and other resources so it's indeed getting all the right versions of everything already.

0
Comment actions Permalink

Dhananjt please attach screenshots of `File | Settings | Build, Execution, Deployment | Toolchains` and `File | Settings | Build, Execution, Deployment | CMake`.

According to "CMake Error at /Applications/CLion.app/Contents/bin/cmake/mac/share/cmake-3.21/Modules/CMakeDetermineCUDACompiler.cmake:113 (message)", it seems CLion uses the local CMake, while it should use the remote one.

Also what is your remote OS? 

0
Comment actions Permalink

Hi,

Indeed it seems like it's using the local one. Unfortunately, the remote CMake's path and my PC's local path are the same.

The Remote server has RHEL 7.9 (Maipo) and I have a macOS.

I tried to change the path to the other CMake path that you see in the screenshot (the CMake I installed in my home folder) but then I get `Unknown argument -f` error in the CMake tab and "Unknown argument -j" while trying to compile and run.

0
Comment actions Permalink

>Unfortunately, the remote CMake's path and my PC's local path are the same.

Could you please elaborate on that? 

In the error message I previously referred to the bundled (local) CMake from `/Applications/CLion.app/Contents/bin/cmake/mac` is mentioned.

0
Comment actions Permalink

The path to CMake is '/usr/bin/make' on both the remote server and my PC.

But if the "Build Tool" config option in Toolchains knows that the path refers to a remote path then it shouldn't be a problem of course.

0
Comment actions Permalink

>The path to CMake is '/usr/bin/make' 

It's path to make, not cmake.

Do you have `/usr/bin/cmake` on the remote machine?

Also please execute `which cmake` in the remote machine and let me know the result.

0
Comment actions Permalink

Right, the path to cmake is what you see in the screenshot. There's no central cmake available from the admin so I'm using my own installation. So the path to CMake is "~/bin/cmake" which will expand to what you see in the screenshot.

0
Comment actions Permalink

Please do `Tools | CMake | Reset Cache and Reload Project` and send me the entire output from the CMake tool window.

0
Comment actions Permalink

This is the output from the Debug tab in CMake:

/Applications/CLion.app/Contents/bin/cmake/mac/bin/cmake -DCMAKE_BUILD_TYPE=Debug -DCMAKE_MAKE_PROGRAM=/Applications/CLion.app/Contents/bin/ninja/mac/ninja -G Ninja -S /Users/dhananjay/Downloads/big_conv -B /Users/dhananjay/Downloads/big_conv/cmake-build-debug
CMake Error at /Applications/CLion.app/Contents/bin/cmake/mac/share/cmake-3.22/Modules/FindPackageHandleStandardArgs.cmake:230 (message):
  Could NOT find EnvModules (missing: EnvModules_COMMAND)
Call Stack (most recent call first):
  /Applications/CLion.app/Contents/bin/cmake/mac/share/cmake-3.22/Modules/FindPackageHandleStandardArgs.cmake:594 (_FPHSA_FAILURE_MESSAGE)
  /Applications/CLion.app/Contents/bin/cmake/mac/share/cmake-3.22/Modules/FindEnvModules.cmake:335 (find_package_handle_standard_args)
  CMakeLists.txt:8 (find_package)


-- Configuring incomplete, errors occurred!

[Failed to reload]

The env errors here are due to some new lines in my CMakeLists.txt:

find_package(EnvModules REQUIRED)
env_module(load CUDA/11.4.2)
env_module(load cuDNN/8.2.1.32-CUDA-11.3.1)

These lines do work as I'm able to build and run the code on the remote server and as another CMake tag shows:

/uio/kant/ifi-ansatt-u07/dhananjt/bin/cmake -DCMAKE_BUILD_TYPE=Debug -DCMAKE_MAKE_PROGRAM=/usr/bin/make -DCMAKE_C_COMPILER=/uio/kant/ifi-ansatt-u07/dhananjt/gcc-9.5.0/bin/gcc -DCMAKE_CXX_COMPILER=/uio/kant/ifi-ansatt-u07/dhananjt/gcc-9.5.0/bin/g++ -G "CodeBlocks - Unix Makefiles" -S /uio/kant/ifi-ansatt-u07/dhananjt/big_conv -B /uio/kant/ifi-ansatt-u07/dhananjt/big_conv/cmake-build-samsida
-- Found EnvModules: /opt/software/lmod/lmod/libexec/lmod  

The following have been reloaded with a version change:
  1) CUDA/11.3.1 => CUDA/11.4.2


The following have been reloaded with a version change:
  1) CUDA/11.4.2 => CUDA/11.3.1

-- The CUDA compiler identification is NVIDIA 11.3.109
-- Detecting CUDA compiler ABI info
-- Detecting CUDA compiler ABI info - done
-- Check for working CUDA compiler: /opt/software/easybuild/software/CUDA/11.3.1/bin/nvcc - skipped
-- Detecting CUDA compile features
-- Detecting CUDA compile features - done
-- Configuring done
-- Generating done
CMake Warning:
  Manually-specified variables were not used by the project:

    CMAKE_CXX_COMPILER
    CMAKE_C_COMPILER


-- Build files have been written to: /uio/kant/ifi-ansatt-u07/dhananjt/big_conv/cmake-build-samsida

[Finished]

I don't think I'll be able to reproduce the exact error I posted weeks ago in this post. My project has changed quite a bit since then. Maybe it's best to close the issue as the errors in the CMake Debug tag aren't causing any real problem for me now.

0
Comment actions Permalink

Ok. You have several CMake profiles in Preferences | Build, Execution, Deployment | CMake. It seems the "Samsida" profile uses the proper remote toolchain called "Samsida", while the "Debug" profile uses the "Default" toolchain.

You need to use the "Samsida" profile, not the "Debug" profile, if you want to build/run/debug the project on the remote machine. It means the "Samsida" profile should be selected in the drop-down - https://www.jetbrains.com/help/clion/quick-cmake-tutorial.html#targets-configs.

0
Comment actions Permalink

Oh ok so it's the default profile throwing those errors. That makes sense. Thank you. 

0
Comment actions Permalink

In the CMake tool window each tab corresponds to a separate CMake profile, please pay attention to the tab's name.

0
Comment actions Permalink

Yeah, I got that. Thank you.

0

Please sign in to leave a comment.