Setting up a CUDA project on a remote server

TLDR:

install from snap with snap install clion --classic, not from flatpak

maybe launch clion with exec=bash -i -c "....."

I had this same problem. I couldn't find a solution anywhere.

I had nvcc on my PATH, set in /etc/environment (as suggested by jetbrains). I configured the CUDAToolkit_ROOT as indicated in the clion docs. I tried setting various paths to nvcc in my project CMakeLists.txt, or in the settings->Build, Execution, Deployment->CMake flags, I tried specifying paths manually under settings->Build, ...->Toolchains - no joy. CMake couldn't find nvcc.

Of course, building the project from terminal worked just fine. It was Clion whom had mucked something royally up. I wanted to use an IDE to debug the project itself (joke's on me), so I persisted.

Clion and Pycharm have a history of clearing and misconfiguring you system environment while launching the application (in my experience - on Ubuntu). Don't ask me how or why or where - I hate this configuration stuff instead of doing actual work, that's why I use an IDE (joke's on me again).

The first hint was under settings->...->Toolchains. I configured CMake: /usr/bin/cmake (my system install). Underneath the godforsaken setting box, Clion said:

(checkmark) Version: 3.22.something

But in my system terminal, I do:

/usr/bin/cmake --version

and get:

cmake version 3.16.3

So Clion was lying, and was still using the built-in CMake version bundled with the Clion application package (somewhere under /var/lib/flatpak/app/com.jetbrains.something/something/bin/something). At this point, everything about setting cmake build flags and environment variables goes out the window. Who knows where those configurations were or weren't applied?

Anyhow, I had had previous success with Pycharm doing a similar environment-clearing-and-buggy behavior. I had gone to the .desktop file which launched Pycharm (under Ubuntu-gnome), and edited the line:

Exec="..."

to be:

Exec=bash -i -c "..."

which would magically preserve my environment variables in Pycharm (the behavior around env vars is still buggy in these IDEs, to be clear).

This fix did not apply to my situation here in 2022 with Clion - flatpak has a .desktop file somewhere deep within its inscrutable path, but the bash -i -c option had no effect here.

I uninstalled the flatpak Clion, and installed the snap version. The Ubuntu software center is too dumb to manage this - because Clion requires classic confinement (wow cool).

To install then, I did:

sudo snap install clion --classic

and magically everything works as expected.

Side note: I’ve also had to put shared-object files built by my projects (*.so) directly onto the system LD_LIBRARY_PATH in order for Pycharm/Clion to find them. Good luck with that. 

Thanks for the detailed reply Chenn.

I should've mentioned that I'm using Mac and I'm running things remotely on a server (edited my question now). I installed CLion by simply downloading it from jetbrains and not via flatpak or snap.

I'm manually pointing it to all the CUDA libraries, CMAKE and other resources so it's indeed getting all the right versions of everything already.

Dhananjt please attach screenshots of `File | Settings | Build, Execution, Deployment | Toolchains` and `File | Settings | Build, Execution, Deployment | CMake`.

According to "CMake Error at /Applications/CLion.app/Contents/bin/cmake/mac/share/cmake-3.21/Modules/CMakeDetermineCUDACompiler.cmake:113 (message)", it seems CLion uses the local CMake, while it should use the remote one.

Also what is your remote OS? 

Hi,

Indeed it seems like it's using the local one. Unfortunately, the remote CMake's path and my PC's local path are the same.

The Remote server has RHEL 7.9 (Maipo) and I have a macOS.

I tried to change the path to the other CMake path that you see in the screenshot (the CMake I installed in my home folder) but then I get `Unknown argument -f` error in the CMake tab and "Unknown argument -j" while trying to compile and run.

>Unfortunately, the remote CMake's path and my PC's local path are the same.

Could you please elaborate on that? 

In the error message I previously referred to the bundled (local) CMake from `/Applications/CLion.app/Contents/bin/cmake/mac` is mentioned.

The path to CMake is '/usr/bin/make' on both the remote server and my PC.

But if the "Build Tool" config option in Toolchains knows that the path refers to a remote path then it shouldn't be a problem of course.

>The path to CMake is '/usr/bin/make' 

It's path to make, not cmake.

Do you have `/usr/bin/cmake` on the remote machine?

Also please execute `which cmake` in the remote machine and let me know the result.

Right, the path to cmake is what you see in the screenshot. There's no central cmake available from the admin so I'm using my own installation. So the path to CMake is "~/bin/cmake" which will expand to what you see in the screenshot.

Please do `Tools | CMake | Reset Cache and Reload Project` and send me the entire output from the CMake tool window.

This is the output from the Debug tab in CMake:

/Applications/CLion.app/Contents/bin/cmake/mac/bin/cmake -DCMAKE_BUILD_TYPE=Debug -DCMAKE_MAKE_PROGRAM=/Applications/CLion.app/Contents/bin/ninja/mac/ninja -G Ninja -S /Users/dhananjay/Downloads/big_conv -B /Users/dhananjay/Downloads/big_conv/cmake-build-debug
CMake Error at /Applications/CLion.app/Contents/bin/cmake/mac/share/cmake-3.22/Modules/FindPackageHandleStandardArgs.cmake:230 (message):
  Could NOT find EnvModules (missing: EnvModules_COMMAND)
Call Stack (most recent call first):
  /Applications/CLion.app/Contents/bin/cmake/mac/share/cmake-3.22/Modules/FindPackageHandleStandardArgs.cmake:594 (_FPHSA_FAILURE_MESSAGE)
  /Applications/CLion.app/Contents/bin/cmake/mac/share/cmake-3.22/Modules/FindEnvModules.cmake:335 (find_package_handle_standard_args)
  CMakeLists.txt:8 (find_package)


-- Configuring incomplete, errors occurred!

[Failed to reload]

The env errors here are due to some new lines in my CMakeLists.txt:

find_package(EnvModules REQUIRED)
env_module(load CUDA/11.4.2)
env_module(load cuDNN/8.2.1.32-CUDA-11.3.1)

These lines do work as I'm able to build and run the code on the remote server and as another CMake tag shows:

/uio/kant/ifi-ansatt-u07/dhananjt/bin/cmake -DCMAKE_BUILD_TYPE=Debug -DCMAKE_MAKE_PROGRAM=/usr/bin/make -DCMAKE_C_COMPILER=/uio/kant/ifi-ansatt-u07/dhananjt/gcc-9.5.0/bin/gcc -DCMAKE_CXX_COMPILER=/uio/kant/ifi-ansatt-u07/dhananjt/gcc-9.5.0/bin/g++ -G "CodeBlocks - Unix Makefiles" -S /uio/kant/ifi-ansatt-u07/dhananjt/big_conv -B /uio/kant/ifi-ansatt-u07/dhananjt/big_conv/cmake-build-samsida
-- Found EnvModules: /opt/software/lmod/lmod/libexec/lmod  

The following have been reloaded with a version change:
  1) CUDA/11.3.1 => CUDA/11.4.2


The following have been reloaded with a version change:
  1) CUDA/11.4.2 => CUDA/11.3.1

-- The CUDA compiler identification is NVIDIA 11.3.109
-- Detecting CUDA compiler ABI info
-- Detecting CUDA compiler ABI info - done
-- Check for working CUDA compiler: /opt/software/easybuild/software/CUDA/11.3.1/bin/nvcc - skipped
-- Detecting CUDA compile features
-- Detecting CUDA compile features - done
-- Configuring done
-- Generating done
CMake Warning:
  Manually-specified variables were not used by the project:

    CMAKE_CXX_COMPILER
    CMAKE_C_COMPILER


-- Build files have been written to: /uio/kant/ifi-ansatt-u07/dhananjt/big_conv/cmake-build-samsida

[Finished]

I don't think I'll be able to reproduce the exact error I posted weeks ago in this post. My project has changed quite a bit since then. Maybe it's best to close the issue as the errors in the CMake Debug tag aren't causing any real problem for me now.

Ok. You have several CMake profiles in Preferences | Build, Execution, Deployment | CMake. It seems the "Samsida" profile uses the proper remote toolchain called "Samsida", while the "Debug" profile uses the "Default" toolchain.

You need to use the "Samsida" profile, not the "Debug" profile, if you want to build/run/debug the project on the remote machine. It means the "Samsida" profile should be selected in the drop-down - https://www.jetbrains.com/help/clion/quick-cmake-tutorial.html#targets-configs.

Oh ok so it's the default profile throwing those errors. That makes sense. Thank you. 

In the CMake tool window each tab corresponds to a separate CMake profile, please pay attention to the tab's name.

Yeah, I got that. Thank you.